DataManager Reference Guide

This guide is for versions Beta 35.1+


Source code

Background Information: A Technology Stack on Top of a (Neo4j) Graph Database

Class DataManager

    For general, high-level database-interaction operations.
    Used by the UI for Page Generation,
    as well as by the web API to produce data for the endpoints.

    This library is primarily a common entry point for data requests: 
    many specific operations get delegated to other, more specialized, libraries.

    This class does NOT get instantiated.
nameargumentsreturns
get_node_labelscls[str]
        Look up and return a list of all the node labels in the database.
        EXAMPLE: ["my_label_1", "my_label_2"]

        :return:    A list of strings
        
nameargumentsreturns
add_new_labelcls, label: strint
        Create a new blank node with the specified label.
        Mostly used for testing.

        :return:    The internal database ID of the new node
        

UTILITIES

nameargumentsreturns
to_int_if_possiblecls, s: strUnion[int, str, None]
        Convert the argument to an integer, if at all possible; otherwise, leave it as a string
        (or leave it as None, if applicable)
        :param s:
        :return:    Either an int or a string
        
nameargumentsreturns
str_to_intcls, s: strint
        Helper function to give more friendly error messages in case non-integers are passed
        in situations where integers are expected.
        Without this function, the user would see cryptic messages such as
        "invalid literal for int() with base 10: 'q123'"

        EXAMPLE of usage:
            try:
                i = cls.str_to_int(i_str)
            except Exception as ex:
                # Do something

        :param s:   A string that should represent an integer
        :return:    The integer represented in the passed string, if applicable;
                        if not, an Exception is raised
        
nameargumentsreturns
experimental_par_dict_cleanercls, data_dict, required_par_list=None, int_list={}dict
        NOT IN CURRENT USE YET
        "Cleaner" for general dictionaries of parameters

        :param data_dict:           A generic dictionary of parameters
        :param required_par_list:   List of names of keys whose presence is required
        :param int_list:            List of names of keys whose values need to be integers
        :return:
        

SCHEMA RELATED

nameargumentsreturns
all_schema_classescls[str]
        Return a list of all the existing Schema classes
        :return:
        
nameargumentsreturns
new_schema_classcls, class_specs: dictNone
        Create a new Schema Class, possibly linked to another existing class,
        and also - typically but optionally - with the special "INSTANCE_OF" link
        to an existing class (often, "Records")
        In case of error, an Exception is raised.

        :param class_specs: A dictionary with the following
                DICTIONARY KEYS:
                new_class_name      The name of the new Class (tolerant of leading/trailing blanks)
                properties_list     The name of all desired Properties, in order
                                    (all comma-separated).  Tolerant of leading/trailing blanks, and of missing property names
                instance_of         Typically, "Records"

                [ALL THE REMAINING KEYS ARE OPTIONAL]
                linked_to           The name of an existing Class node, to link to
                rel_name            The name to give to the above relationship
                rel_dir             The relationship direction, from the point of view of the newly-added node

        :return:            None
        
nameargumentsreturns
add_schema_relationship_handlercls, class_specs: dictNone

        In case of error, an Exception is raised.

        :param class_specs: A dictionary with the following
                DICTIONARY KEYS:
                    from_class_name
                    to_class_name
                    rel_name

        :return: None
        
nameargumentsreturns
schema_add_property_to_class_handlercls, specs_dict: dictNone
        Add a new Property to an existing Classes

        In case of error, an Exception is raised.

        :param specs_dict: A dictionary with the following
                DICTIONARY KEYS:
                    prop_name       (any leading/trailing blanks are ignored)
                    class_name      (any leading/trailing blanks are ignored)

        :return: None
        
nameargumentsreturns
delete_schema_relationship_handlercls, class_specs: dictNone
        Delete the relationship(s) with the specified name
        between the 2 existing Class nodes (identified by their respective names),
        going in the from -> to direction direction.

        In case of error, an Exception is raised.

        :param class_specs: A dictionary with the following
                DICTIONARY KEYS:
                    from_class_name
                    to_class_name
                    rel_name

        :return: None
        
nameargumentsreturns
add_data_relationship_handlercls, data_dict: dictNone
        Add the specified relationship (edge) between data nodes.
        In case of error, an Exception is raised.

        :param data_dict: A dictionary with the following
                from                    The uri of the node from which the relationship originates
                to                      The uri of the node into which the relationship takes
                rel_name                The name of the relationship to add
                schema_code (optional)  If passed, the appropriate plugin gets invoked

        :return: None
        
nameargumentsreturns
remove_data_relationship_handlercls, data_dict: dictNone
        Remove the specified relationship (edge) between data nodes.
        In case of error, an Exception is raised.

        :param data_dict: A dictionary with the following
                DICTIONARY KEYS:
                    from                    The uri of the node from which the relationship originates
                    to                      The uri of the node into which the relationship takes
                    rel_name                The name of the relationship to remove
                    schema_code (optional)  If passed, the appropriate plugin gets invoked

        :return: None
        
nameargumentsreturns
get_leaf_recordscls[str]
        Get all Classes that are, directly or indirectly, INSTANCE_OF the Class "Records",
        as long as they are leaf nodes (with no other Class that is an INSTANCE_OF them.)

        EXAMPLE: if the "Foreign Vocabulary" Class is an INSTANCE_OF the Class "Records",
                 and if "French Vocabulary" and "German Vocabulary" are instances of "Foreign Vocabulary",
                 then "French Vocabulary" and "German Vocabulary" (but NOT "Foreign Vocabulary")
                 would be returned

        :return: A list of strings with the Class names
                 EXAMPLE:
                    ["Cuisine Type","Entrees","French Vocabulary","German Vocabulary","Restaurants","Site Link"]
        
nameargumentsreturns
get_text_media_contentcls, uri :str, schema_code, public_required = Falsestr
        Fetch and return the contents of a media item stored on a local file,
        optionally requiring it to be marked as "public".
        In case of error, raise an Exception

        :param uri:             A string identifying the desired Content Item, which ought to be text media
        :param schema_code:     TODO: maybe phase out
        :param public_required: If True, the Content Item is returned only if has an the attribute "public: true"

        :return:                A string with the HTML text of the requested note;
                                    or an Exception in case of failure
                                    (e.g., if public_required is True and the item isn't public)

        
nameargumentsreturns
get_records_by_linkcls, request_data: dict[dict]
        Locate and return the data of the nodes linked to the one specified by uri,
        by the relationship named by rel_name, in the direction specified by dir

        :param request_data: A dictionary with 3 keys, "uri", "rel_name", "dir"
        :return:             A list of dictionaries with all the properties of the neighbor nodes
        
nameargumentsreturns
get_link_summarycls, uri :str, omit_names = Nonedict
        Return a dictionary structure identifying the names and counts of all
        inbound and outbound links to/from the given data node.
        TODO: move most of it to the "~ FOLLOW LINKS ~" section of NeoAccess

        :param uri:         String with the URI of a data node
        :param omit_names:  Optional list of relationship names to disregard
        :return:            A dictionary with the names and counts of inbound and outbound links.
                            Each inner list is a pair [name, count]
                            EXAMPLE:
                                {
                                    "in": [
                                        ["BA_served_at", 1]
                                    ],
                                    "out": [
                                        ["BA_located_in", 1],
                                        ["BA_cuisine_type", 2]
                                    ]
                                }
        
nameargumentsreturns
update_content_itemcls, post_data: dictNone
        Update an existing Content Item.
        In case of error, an Exception is raised

        NOTE: the "schema_code" field is currently required, but it's redundant.  Only
              used as a safety mechanism against incorrect values of the URI
              (TODO: maybe ditch, or use the Class name instead)

        :return:    None.  In case of error, an Exception is raised
        
nameargumentsreturns
delete_content_itemcls, uri: str, schema_code: strNone
        Delete the specified individual Content Item.
        Note that schema_code is redundant.
        In case of error, an Exception is raised

        :param uri:         String version of the unique ID
        :param schema_code: Redundant
        :return:            None.  In case of error, an Exception is raised
        
nameargumentsreturns
new_content_item_in_categorycls, post_data: dictstr
        Create a new Content Item attached to a particular Category,
        at a specified position on the Category page

        :param post_data:   A dict containing the following keys
            - "category_id"  (for the linking to a Category)
            - Schema-related keys:
                    * schema_code (Required)
                    * schema_uri (Optional)
                    * class_name (Required only for Class Items of type "record")

            - insert_after        Either a URI of an existing Content Item attached to this Category,
                                  or one of the special values "TOP" or "BOTTOM"
            - *PLUS* all applicable plugin-specific fields (all the key/values for the new Content Item)

        :return:    The URI of the newly-created node.
                    In case of error, an Exception is raised
        
nameargumentsreturns
lookup_media_recordcls, uri: intUnion[dict, None]
        Attempt to retrieve the metadata for the media file attached to the specified Content Item
        TODO: move to MediaManager class

        :param uri: An integer with the URI of the Content Item
        :return:        If found, return a dict with the record; otherwise, return None
        

SEARCH

nameargumentsreturns
search_for_wordcls, word: str[dict]
        Look up any stored words that contains the requested string
        (ignoring case and leading/trailing blanks.)

        Then locate the Content nodes that are indexed by any of those words.
        Return a (possibly empty) list of the data of all the found nodes.

        :param word:    A string, typically containing a word or word fragment;
                            case and leading/trailing blanks are ignored
        :return:
        
nameargumentsreturns
extract_website_titlecls, url :strstr
        Retrieve the Title of a remote webpage, given its URL

        :param url: URL of the website whose title we want to fetch.
                        EXAMPLE:  "https://brainannex.org"
        :return:    The "Title" of the website.
                        In case unable to locate the web page, or unable to extract its Title,
                        raise an Exception
        

FILTERS

nameargumentsreturns
get_nodes_by_filtercls, filter_dict[dict]

        :param filter_dict: A dictionary.
                            EXAMPLE: {"labels": "BA", "key_name": "uri", "key_value": 123, "limit": 25}

        :return:            A (possibly-empty) list of dictionaries
        

IMPORT EXPORT

nameargumentsreturns
export_full_dbasecls

nameargumentsreturns
upload_import_jsoncls, verbose=False, return_url=Nonestr
        Modify the database, based on the contents of the uploaded file (expected to contain the JSON format
        of a Neo4j export)

        :param verbose:
        :param return_url:
        :return:            Status string (error or success message)
        
nameargumentsreturns
upload_import_json_filecls, post_pars, verbose=Falsestr
        Manage the upload and import into the database of a data file in JSON format.

        :return:    Status string, if successful.  In case of error, an Exception is raised
        
nameargumentsreturns
data_intake_statuscls

nameargumentsreturns
do_stop_data_intakeclsNone
        Request that the continuous data import cease upon the completion of the current import

        :return:    None
        
nameargumentsreturns
do_bulk_importcls, intake_folder: str, outtake_folder: str, schema_class: strstr
        Bulk-import all the JSON files in the intake_folder directory.
        The import will continue until the folder is empty,
        or until the cls.ongoing_data_intake property is set to False

        Failure of individual imports is logged, but does not terminate the operation.
        An Exception is raised if any of the following happens:
            * The log file cannot be accessed/created
            * Any of the data files cannot be moved to their final destination

        :param intake_folder:   Name of folder (ending with "/") where the files to import are located
        :param outtake_folder:
        :param schema_class:
        :return: 
        
nameargumentsreturns
process_file_to_importcls, f: str, intake_folder: str, outtake_folder: str, schema_class: strNone
        Import a JSON file, located in a particular folder, and then move it to a designated folder.
        Keep a log of the operations.
        Exceptions are caught and logged; if the Exceptions involves moving the process file, a new one is raised

        :param f:               Name of file to import
        :param intake_folder:   Name of folder (ending with "/") where the files to import are located
        :param outtake_folder:
        :param schema_class:
        :return:                None
        
nameargumentsreturns
archive_data_filecls, src_fullname, dest_fullname, outtake_folderNone
        Move the processed file to its final location.
        In case of error, the Exception is caught, logged and re-raised

        :param src_fullname:
        :param dest_fullname:
        :param outtake_folder:
        :return:                None
        
nameargumentsreturns
import_datafilecls, basename, full_filename, test_only=Truestr
        TODO: NOT in current use.  See DocumentationGenerator.import_python_file()
        TODO: generalize!  For now, used for an ad-hoc import, using REGEX to extract the desired fields

        :param basename:        EXAMPLE: "my_file_being_uploaded.txt"
        :param full_filename:   EXAMPLE: "D:/tmp/my_file_being_uploaded.txt"
        :param test_only:       If True, the file is parsed, but nothing is actually added to the database

        :return:                String with status message (whether successful or not)
        
nameargumentsreturns
define_patternclsstr
        Define a REGEX pattern for parsing of data files, for use in import_datafile()

        The pattern is expected to be used in a re.findall() that uses re.DOTALL as the last argument

        :return:    A string with a REGEX pattern
        

LOGGING

nameargumentsreturns
init_logfileclsNone
        Prepare a handle for the log file

        :return:
        
nameargumentsreturns
append_to_logcls, msgNone

        :param msg:
        :return: