Meta Data service

FOXDEN Meta Data service is responsible for handling meta-data records. The meta-data records are stored based on pre-defined schema file. An example of schema files you can find here.

Here the main APIs:

• GET methods to retrieve meta-data information:
  • /meta end-point provides list of schema unit maps, each map contains schema attribute and its units key-value pairs, no authorization is required
  • /record end-point provides meta-data record for given did, no authorization is required
  • /records end-point provides all meta-data records for given query, requires valid authorization token
  • / main end-point to search for meta-data
  • /tmpl/record fetches template record
  • /updatedoi updates doi information in metadata record
• POST end-points:
  • / main end-point in inject meta-data record, required write authorization scope
  • /search end-point to search for meta-data records based on Query Language, requires valid authorization token
  • /count end-point provides number of documents for provided query,
  • /tmpl/submit/record submits template record requires valid authorization token
• PUT end-points:
  • / updates metadata record
  • /tmpl/record updates template record
• DELETE end-points:
  • /record end-point allows delete meta-data record, requires delete authorization token
  • /tmp/record end-point deletes template record

Examples of using meta-data service APIs

# record.json can be one of CHESS meta-data records

# inject new record
curl -v -X POST -H "Content-type: application/json" \
    -H "Authorization: Bearer $token" \
    -d@./record.json \
    http://localhost:8300

# perform search with pagination
curl -X POST \
    -H "Authorization: bearer $token" \
    -H "Content-type: application/json" \
    -d '{"client":"go-client","service_query":{"query":"{}","spec":null,"sql":"","idx":0,"limit":2}}' \
    http://localhost:8300/search

# Retrieve concrete record with did=/beamline=4b/btr=btr-123/cycle=myc/samples_name=ms
# we need to properly encode did parameter, i.e. replace / with %2F and = with %3D
# User should rely on specific library to do this, e.g. in python
#
# from urllib.parse import urlencode
# did = "/beamline=4b/btr=btr-123/cycle=myc/samples_name=ms"
# params = {"did": did}
# url = "http://localhost:8300?" + urlencode(params)

curl -H "Accept: application/json" \
    -H "Authorization: bearer $token" \
    http://localhost:8300?did=%2Fbeamline%3D4b%2Fbtr%3Dbtr-123%2Fcycle%3Dmyc%2Fsamples_name%3Dms

Backends

The Metadata services supports setup of different backends:

• MongoDB: document oriented database used by Metadata service by default. Since it is external project you need to properly install it on your system. In addition we highly recommend to setup the following indexes within MongoDB:

# In MongoDB it is desired to create individual indexes
# to speed up data look-up.

# if your database is called foxden.meta, then setup your indexes as following:
# Step 1: login to your MongoDB

# Step 2: use your desired database, e.g. foxden
mongo > use foxden

# Step 3: create necessary indexes

# index on did should be unique
foxden> db.meta.createIndex({did:1}, {unique: true})

# index on doi is not unique as we may have records with no DOI
foxden> db.meta.createIndex({doi:1})

# verify your indexes
foxden> db.meta.getIndexes()

• Badger: embedded key-value database developed by Dgraph
• Clover: a lightweight document-oriented NoSQL database written in pure Golang.
• Tiedot: a basic document (NoSQL) database in Go.