Using Eve's DB layer without HTTP

Question

In my application, the MongoDB collections need to be updated by a server-side script job (IE: a cron job that scrapes/pulls from other APIs every 30minutes). What I really want to do is make updates to the MongoDB collections, but have the data be validated against the schema and include metadata (updated, created, etc).

The two ways that come to mind to solve this is:

1. Have a fake client to do HTTP POST/PUT/PATCHES. However, this means this fake client would have to deal with things like authentication/authorization/last-modified-since.
2. Use PyMongo to interact with the DB directly. However, this means I wouldn't have the data validation, or the metadata stored.

Does Eve have hooks for the database so that I can do Eve-rich database updates without HTTP?

Purlin

Answer 1

I was able to run this in a separate script that can be run periodically by jenkins. The app in run.py that I'm importing is the one I had by the end of the eve quickstart.

from run import app
from eve.methods.post import post_internal

payload = {
    "firstname": "Ray",
    "lastname": "LaMontagne",
    "role": ["contributor"]
}

with app.test_request_context():
    x = post_internal('people', payload)
    print(x)

post_internal runs eve.utils.parse_request, which relies on flask.request, so with app.test_request_context() is required. app.app_context() isn't sufficient for this method.

Read the docs for appcontext and reqcontext if you're new to flask (like me).

Purlin

put_internal, patch_internal, deleteitem_internal are now available. – Purlin

Answer 2

As of v0.5 (currently on the development branch but you can pull and use it right away) you can use post_internal for adding data:

   Intended for internal post calls, this method is not rate limited,
   authentication is not checked and pre-request events are not raised.
   Adds one or more documents to a resource. Each document is validated
   against the domain schema. If validation passes the document is inserted
   and ID_FIELD, LAST_UPDATED and DATE_CREATED along with a link to the
   document are returned. If validation fails, a list of validation issues
   is returned.

It would probably make sense to add more internal methods to cover all CRUD operation which are now available via HTTP. You can still invoke them right away, though.

Update: v0.5 has been released with _internal methods available for all CRUD operations.

Rook

Brilliant! I may follow the post vs post_internal pattern for put/patch/delete. – Purlin

2 follow-up questions: 1) Any suggestions for interacting with the flask/eve from a separate .py script? This answer seems like it would work, but don't like the idea of creating an http endpoint just for an internal task, and then putting in logic to keep others out. 2) Creating put_internal the way post_internal was created won't be enough because of the etag/IF_MATCH constraint that takes place in get_document. – Purlin

On 2) you can experiment with disabling etag matching via configuration (even temporarily). I plan on looking into it and see if I can come up with new 'internal' edit methods. On 1) never tried that but it would work if you're hitting the webserver of course; or you could use greenlet/celery something like that i guess? Don't think flask support that natively (not my field though). – Rook

Yeah I thought 1) might be more of a boarder flask question. as for 2), perhaps implementing the etag comparison as a decorator would make it easy to bypass with internal_put/delete/patch. – Purlin

Answer 3

In case you wanted to do the following in a custom endpoint...

1. accept some POSTed data
2. manipulate the data in some way
3. perform a post_internal()
4. return the result as a response

...you'd do it something like this:

from eve.methods.post import post_internal
from eve.render import send_response

def my_custom_endpoint(**kwargs):

    data = json.loads(request.data.decode())

    # <manipulate data here>    

    resp = post_internal('crew', data)
    return send_response('crew', resp)

In reality you're probably better off to use Eve's Event Hooks to do this sort of thing. But in case there were some situation that Event Hooks didn't cover, this approach might be useful.

Bole