DataManager Reference Guide
This guide is for versions Beta 47+
Source code
Background Information:
A Technology Stack on Top of a (Neo4j) Graph Database
For general, high-level database-interaction operations.
Used by the UI for Page Generation,
as well as by the web API to produce data for the endpoints.
This library is primarily a common entry point for data requests:
many specific operations get delegated to other, more specialized, libraries.
Static class that does NOT get instantiated;
however, it must be initialized with a call to set_database()
| name | arguments | returns |
|---|---|---|
| set_database | db :NeoAccess | None |
IMPORTANT: this method MUST be called before using this class!
:param db: Database-interface object, created with the NeoAccess library
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| get_node_labels | [str] | |
Look up and return a list of all the node labels in the database.
EXAMPLE: ["my_label_1", "my_label_2"]
:return: A list of strings
|
||
| name | arguments | returns |
|---|---|---|
| add_new_label | label: str | int |
Create a new blank node with the specified label.
Mostly used for testing.
:return: The internal database ID of the new node
|
||
| name | arguments | returns |
|---|---|---|
| to_int_if_possible | s: str | |
Convert the argument to an integer, if at all possible; otherwise, leave it as a string
(or leave it as None, if applicable)
:param s: Value to convert to integer, if possible
:return: Either an int version of the passed value, or that same value
|
||
| name | arguments | returns |
|---|---|---|
| str_to_int | s: str | int |
Helper function to give more friendly error messages in case non-integers are passed
in situations where integers are expected.
Without this function, the user would see cryptic messages such as
"invalid literal for int() with base 10: 'q123'"
EXAMPLE of usage:
try:
i = cls.str_to_int(i_str)
except Exception as ex:
# Do something
:param s: A string that should represent an integer
:return: The integer represented in the passed string, if applicable;
if not, an Exception is raised
|
||
| name | arguments | returns |
|---|---|---|
| all_schema_classes | [str] | |
Return a list of all the existing Schema classes
:return:
|
||
| name | arguments | returns |
|---|---|---|
| get_schema_visualization_data | ||
Create and return an object with all the info
to visualise a graph with the database Schema info
:return: An object of class PyGraphScape
|
||
| name | arguments | returns |
|---|---|---|
| new_schema_class | class_specs: dict | None |
Create a new Schema Class, possibly linked to another existing class,
and also - typically but optionally - with the special "INSTANCE_OF" link
to an existing class (often, "Records")
In case of error, an Exception is raised.
:param class_specs: A dictionary with the following
DICTIONARY KEYS:
new_class_name The name of the new Class (tolerant of leading/trailing blanks)
properties_list The name of all desired Properties, in order
(all comma-separated). Tolerant of leading/trailing blanks, and of missing property names
instance_of Typically, "Records"
[ALL THE REMAINING KEYS ARE OPTIONAL]
linked_to The name of an existing Class node, to link to
rel_name The name to give to the above relationship
rel_dir The relationship direction, from the point of view of the newly-added node
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| add_schema_relationship_handler | class_specs: dict | None |
In case of error, an Exception is raised.
:param class_specs: A dictionary with the following
DICTIONARY KEYS:
from_class_name
to_class_name
rel_name
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| schema_add_property_to_class_handler | specs_dict: dict | None |
Add a new Property to an existing Classes
In case of error, an Exception is raised.
:param specs_dict: A dictionary with the following
DICTIONARY KEYS:
prop_name (any leading/trailing blanks are ignored)
class_name (any leading/trailing blanks are ignored)
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| delete_schema_relationship_handler | class_specs: dict | None |
Delete the relationship(s) with the specified name
between the 2 existing Class nodes (identified by their respective names),
going in the from -> to direction direction.
In case of error, an Exception is raised.
:param class_specs: A dictionary with the following
DICTIONARY KEYS:
from_class_name
to_class_name
rel_name
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| add_data_relationship_handler | data_dict: dict | None |
Add the specified relationship (edge) between data nodes.
In case of error, an Exception is raised.
:param data_dict: A dictionary with the following
from The uri of the node from which the relationship originates
to The uri of the node into which the relationship takes
rel_name The name of the relationship to add
schema_code (optional) If passed, the appropriate plugin gets invoked
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| remove_data_relationship_handler | data_dict: dict | None |
Remove the specified relationship (edge) between data nodes.
In case of error, an Exception is raised.
:param data_dict: A dictionary with the following
DICTIONARY KEYS:
from The uri of the node from which the relationship originates
to The uri of the node into which the relationship takes
rel_name The name of the relationship to remove
schema_code (optional) If passed, the appropriate plugin gets invoked
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| get_leaf_records | [str] | |
Get all Classes that are, directly or indirectly, INSTANCE_OF the Class "Records",
as long as they are leaf nodes (with no other Class that is an INSTANCE_OF them.)
EXAMPLE: if the "Foreign Vocabulary" Class is an INSTANCE_OF the Class "Records",
and if "French Vocabulary" and "German Vocabulary" are instances of "Foreign Vocabulary",
then "French Vocabulary" and "German Vocabulary" (but NOT "Foreign Vocabulary")
would be returned
:return: A list of strings with the Class names
EXAMPLE:
["Cuisine Type","Entrees","French Vocabulary","German Vocabulary","Restaurants","Site Link"]
|
||
| name | arguments | returns |
|---|---|---|
| get_text_media_content | uri :str, public_required = False | str |
Fetch and return the contents of a media item stored on a local file,
optionally requiring it to be marked as "public".
:param uri: A string identifying the desired Content Item, which ought to be text media
:param public_required: If True, the Content Item is returned
only if its database node has an the attribute "public: true"
:return: A string with the HTML text of the requested note;
or an Exception in case of failure
(e.g., if public_required is True and the item isn't public)
|
||
| name | arguments | returns |
|---|---|---|
| get_records_by_class | class_name :str, field_name :str, order_by :str | [] |
Return a list of values of a particular field, of all the records of the given Class,
optionally sorted by the given field
:param class_name:
:param field_name:
:param order_by:
:return: A list of values
|
||
| name | arguments | returns |
|---|---|---|
| get_records_by_link | request_data: dict | [dict] |
Locate and return the data of the nodes linked to the one specified by uri,
by the relationship named by rel_name, in the direction specified by dir
:param request_data: A dictionary with 3 keys, "uri", "rel_name", "dir"
:return: A list of dictionaries with all the properties of the neighbor nodes
|
||
| name | arguments | returns |
|---|---|---|
| get_link_summary | uri :str, omit_names = None | dict |
Return a dictionary structure identifying the names and counts of all
inbound and outbound links to/from the given data node.
:param uri: String with the URI of a data node
:param omit_names: Optional list of relationship names to disregard
:return: A dictionary with the names and counts of inbound and outbound links.
Each inner list is a pair [name, count]
EXAMPLE:
{
"in": [
["BA_served_at", 1]
],
"out": [
["BA_located_in", 1],
["BA_cuisine_type", 2]
]
}
|
||
| name | arguments | returns |
|---|---|---|
| update_content_item | uri :str, class_name :str, update_data: dict | None |
Update an existing Content Item.
No harm if new values are identical to the earlier old values.
Note: class_name is redundant
Notes:
- if a field is blank, it gets completely dropped from the node
- leading/trailing blanks in the field values are stripped away
:param uri: String with a unique identifier for the Content Item to update
:param class_name: Name of the Schema Class of the Content Item
:param update_data: A dict of data field names and their desired new values
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| delete_content_item | uri: str, class_name: str | None |
Delete the specified individual Content Item.
Note: class_name is redundant; used as a safety mechanism
against incorrect values of their uri
:param uri: String version of the unique ID
:param class_name: Name of the Schema Class of the Content Item
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| create_data_node | class_name :str, item_data: dict | dict |
Create a new Data Node.
RESTRICTION: currently, not to be used for any Content Item that
requires plugin-specific actions
:param class_name:
:param item_data:
:return: A dict with the internal database ID and uri
assigned to the newly-created node
EXAMPLE: {"internal_id": 123, "uri": "rs-8"}
|
||
| name | arguments | returns |
|---|---|---|
| add_new_content_item_to_category | category_uri :str, class_name :str, insert_after :str, item_data: dict | str |
Create a new Content Item attached to a particular Category,
at a specified position on the Category page
NOTE: this is a newer version of new_content_item_in_category()
:param category_uri: String to identify the Category that we're linking to
:param class_name: Name of the Class of the new Content Item
:param insert_after: Either the URI of an existing Content Item attached to this Category,
or one of the special values "TOP" or "BOTTOM"
:param item_data: Dict with all applicable plugin-specific fields (all the key/values for the new Content Item)
:return: The URI of the newly-created Data Node
|
||
| name | arguments | returns |
|---|---|---|
| new_content_item_in_category | post_data: dict | str |
NOTE: this method will be phased out in favor of add_new_content_item_to_category()
Create a new Content Item attached to a particular Category,
at a specified position on the Category page
:param post_data: A dict containing the following keys
- "category_id" (for the linking to a Category)
- Schema-related keys:
* schema_uri (Optional)
* class_name (Required only for Class Items of type "record")
- insert_after Either the URI of an existing Content Item attached to this Category,
or one of the special values "TOP" or "BOTTOM"
- *PLUS* all applicable plugin-specific fields (all the key/values for the new Content Item)
:return: The URI of the newly-created Data Node.
In case of error, an Exception is raised
|
||
| name | arguments | returns |
|---|---|---|
| new_content_item_in_category_final_step | insert_after :str, category_id :str, new_uri, class_name, post_data, original_post_data | |
| name | arguments | returns |
|---|---|---|
| switch_category | data_dict | None |
Switch one or more Content Items from being attached to a given Category,
to another one
:param data_dict: Dict with 3 keys:
items list of string URI's of Content Items
to relocate across Categories
from URI of the old Category
to URI of the new Category
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| search_for_terms | words :str, search_category="" | ([dict], str) |
Carry out a full-text search for a word, or set of words - possibly restricted to some Categories
:param words: String containing one or more words to search for
:param search_category: (OPTIONAL) URI of Category. If supplied, all searching will
be limited to Content Items in this Category
or in any of its sub-categories
:return: A pair consisting of:
1) list of dictionaries, each with the record data of a search result
2) a string with a caption to describe these search results
|
||
| name | arguments | returns |
|---|---|---|
| search_for_word | word :str, search_category="" | [dict] |
Look up any stored words that contains the requested string
(ignoring case and leading/trailing blanks.)
Then locate the Content nodes that are indexed by any of those words.
Return a (possibly empty) list of the data of all the found nodes.
:param word: A string, typically containing a word or word fragment;
case and leading/trailing blanks are ignored
:param search_category: (OPTIONAL) URI of Category. If supplied, all searching will
be limited to Content Items in this Category
or in any of its sub-categories
:return: A list of dictionaries, each with the record data of a search result
|
||
| name | arguments | returns |
|---|---|---|
| search_for_all_words | word_list :[str], search_category="" | [dict] |
Look up any stored words that contains the requested string
(ignoring case and leading/trailing blanks.)
Then locate the Content nodes that are indexed by any of those words.
Return a (possibly empty) list of the data of all the found nodes.
:param word_list: A list of strings, each typically containing a word or word fragment;
case and leading/trailing blanks are ignored
:param search_category: (OPTIONAL) URI of Category. If supplied, all searching will
be limited to Content Items in this Category
or in any of its sub-categories
:return: A list of dictionaries, each with the record data of a search result
|
||
| name | arguments | returns |
|---|---|---|
| extract_website_title | url :str | str |
Retrieve the Title of a remote webpage, given its URL
:param url: URL of the website whose title we want to fetch.
EXAMPLE: "https://brainannex.org"
:return: The "Title" of the website.
In case unable to locate the web page, or unable to extract its Title,
raise an Exception
|
||
| name | arguments | returns |
|---|---|---|
| get_nodes_by_filter | filter_dict :dict | [dict] |
Return the list of the database nodes that match all the requirements spelled out in the given filter
:param filter_dict: A dictionary, with keys:
"label" The name of a node label
"class_name" NOT CURRENTLY IMPLEMENTED
"key_name" A string with the name of a node attribute;
if provided, key_value must be passed, too
"key_value" The required value for the above key; if provided, key_name must be passed, too.
Note: no requirement for the key to be primary
"clause" NOT CURRENTLY IMPLEMENTED
MUST use "n" as dummy name.
EXAMPLE: "n.name CONTAINS 'art'"
"order_by" Field name, or comma-separated list;
each name may optionally be followed by "DESC"
NOTE: if ordering by a non-existing field, "skip" may not work as expected;
this seems to be a Cypher/Neo4j bug
"skip" The number of initial entries (in the context of specified order) to skip
"limit" The max number of entries to return
EXAMPLES:
{"label": "BA", "key_name": "uri", "key_value": "sl-123"}
{"label": "doctor", "limit": 25, "skip": 50}
{'label': 'YouTube Channel', 'clause': "n.name CONTAINS 'sc'", 'order_by': 'name'}
{'label': 'Quote', 'clause': "n.quote CONTAINS 'kiss'", 'order_by': 'attribution,quote'}
:return: A (possibly-empty) list of dictionaries; each dict contains the data for a node
|
||
| name | arguments | returns |
|---|---|---|
| export_full_dbase | dict | |
Export the entire Neo4j database as a JSON string.
IMPORTANT: APOC must be activated in the database, to use this function.
Otherwise it'll raise an Exception
EXAMPLE:
{ 'nodes': 2,
'relationships': 1,
'properties': 6,
'data': '[{"type":"node","id":"3","labels":["User"],"properties":{"name":"Adam","age":32,"male":true}},\n
{"type":"node","id":"4","labels":["User"],"properties":{"name":"Eve","age":18}},\n
{"id":"1","type":"relationship","label":"KNOWS","properties":{"since":2003},"start":{"id":"3","labels":["User"]},"end":{"id":"4","labels":["User"]}}\n
]'
}
:return: A dictionary specifying the number of nodes exported ("nodes"),
the number of relationships ("relationships"),
and the number of properties ("properties"),
as well as a "data" field with the actual export as a JSON string
|
||
| name | arguments | returns |
|---|---|---|
| upload_import_json | files, upload_dir :str, return_url=None, verbose=False | str |
Modify the database, based on the contents of the uploaded file (expected to contain the JSON format
of a Neo4j export)
:param files: An ImmutableMultiDict object.
EXAMPLE: ImmutableMultiDict([('imported_datafile',
|
||
| name | arguments | returns |
|---|---|---|
| upload_import_json_file | files, upload_dir :str, post_pars, verbose=False | str |
Manage the upload and import into the database of a data file in JSON format.
:param files: An ImmutableMultiDict object.
EXAMPLE: ImmutableMultiDict([('imported_datafile',
|
||
| name | arguments | returns |
|---|---|---|
| data_intake_status | ||
| name | arguments | returns |
|---|---|---|
| do_stop_data_intake | None | |
Request that the continuous data import cease upon the completion of the current import
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| do_bulk_import | intake_folder: str, outtake_folder: str, schema_class: str | str |
Bulk-import all the JSON files in the intake_folder directory.
The import will continue until the folder is empty,
or until the cls.ongoing_data_intake property is set to False
Failure of individual imports is logged, but does not terminate the operation.
An Exception is raised if any of the following happens:
* The log file cannot be accessed/created
* Any of the data files cannot be moved to their final destination
:param intake_folder: Name of folder (ending with "/") where the files to import are located
:param outtake_folder:
:param schema_class:
:return:
|
||
| name | arguments | returns |
|---|---|---|
| process_file_to_import | f: str, intake_folder: str, outtake_folder: str, schema_class: str | None |
Import a JSON file, located in a particular folder, and then move it to a designated folder.
Keep a log of the operations.
Exceptions are caught and logged; if the Exceptions involves moving the process file, a new one is raised
:param f: Name of file to import
:param intake_folder: Name of folder (ending with "/") where the files to import are located
:param outtake_folder:
:param schema_class:
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| archive_data_file | src_fullname, dest_fullname, outtake_folder | None |
Move the processed file to its final location.
In case of error, the Exception is caught, logged and re-raised
:param src_fullname:
:param dest_fullname:
:param outtake_folder:
:return: None
|
||
| name | arguments | returns |
|---|---|---|
| define_pattern | str | |
Define a REGEX pattern for parsing of data files, for use in import_datafile()
The pattern is expected to be used in a re.findall() that uses re.DOTALL as the last argument
:return: A string with a REGEX pattern
|
||
| name | arguments | returns |
|---|---|---|
| init_logfile | None | |
Prepare a handle for the log file
:return:
|
||
| name | arguments | returns |
|---|---|---|
| append_to_log | msg | None |
:param msg:
:return:
|
||