====== Using the Pacific EMIS REST API ====== This page documents the Pacific EMIS REST API (or RESTful API). There are a few things to note as convention for these docs. * '''' is used to designate the base URL of the EMIS system (e.g. fedemis.doe.fm for FSM). * Endpoints typically will start with ''https'' which is preferred. However, it may be some contexts only support ''http''. ===== Logging In ===== Login Endpoint: https:///api/token Pass the user account name and password as form-urlencoded data, like so: POST https:///api/token HTTP/1.1 Host: Accept: application/json, text/plain, / Origin: https:// Content-Type: application/x-www-form-urlencoded Accept-Encoding: gzip, deflate, br Accept-Language: en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7 ....(other headers may be present by default) grant_type=password&username=yourusername&password=yourpassword On successful login (status 200) you will receive a JSON package in reply. Most of this information is relevant to the Pacific EMIS web application front end (i.e. configuration of menu, permissions etc.) but the most important thing here is the access_token. In all subsequent calls to the EMIS, add the authorization header type **bearer** using this token: Host: Connection: keep-alive Content-Length: 56 Accept: application/json, text/plain, / Origin: https:// Authorization: Bearer j7Yqqk4vVOZT8qGZ3Nt09Nv9XXYCk6hKWtoLra7CBJZZ0YZqq17PrlO0c0TCc2knuUzKLz4H96BHID33uqrJvkUJ7V16qNA5iZbSgAry6_Om3WL-fUELm5AOpYpdiAdvo8qNWBe (....etc) If the token times out, you will get a 401 Unauthorized status. You should go through the login again to get a fresh token. ===== Lookup Lists ===== You may find it useful to have lookup lists available for interactive selection of data. Lookup Endpoint: https:///api/lookups/collection/core This returns a JSON object in which each property is an array of ‘code/name’ objects. Each of these objects has a property C – the code, and N – the ‘name’ or description for that code. Where there is a hierarchy of lookups, a list may include an additional property. For example, authorities has a property T, which points to authorityTypes; authorityTypes in turn has property G which points to authorityGovt. For example, see this screen grab from fiddler {{ :rest-api:rest-api-xml-10.jpg?nolink |}} The codeset ‘accreditationTerms’ contains the names associated to each standard and criteria in the accreditation survey. Here’s a section: {{ :rest-api:rest-api-xml-11.jpg?nolink |}} In each item there is: * **C** – the code of the standard or criteria * **N** – the name associated to that code. This is in fact pulled directly from a template copy of the survey. * **S** – ‘a short name’ I found it convenient to introduce a ‘short name’ for each standard for display purposes. You may wish to use this one. * **T** – type . this is S for a standard, and C for a criteria. So to get either a long or short description, find the record in accreditationTerms by searching for the code (e.g. SE.1) and Read back the N or S property. You could for example in javascript do this with lodash : _.find(lookups.accreditationTerms, r => r.C == ‘SE.1’).N for example perhaps using the code table gives you additional ways of iterating over the standards or criteria. ==== Vocab lookup ==== A special type of lookup you get when loading the app or starting to use the REST API is the "vocab". The vocab, short for vocabularies, is a means to localize the Pacific EMIS terminologies. Different countries have different terminologies, for example: * Districts in FSM are States, in RMI they are Atolls/Islands and in Solomon Islands they are provinces, or * Grades vs Grade Levels vs Class Levels While referred to differently in different countries, they are in essence the same things. So the Pacific EMIS uses one common term in the source code but in the UI it can easily be translated to the country's specific preferred terminology. For example, in the source code developers would work with "District" but in the web user interfaces the word District would translate to the country specific terminology (State in FSM, Atoll/Island in RMI, etc.). In the current web UI, this is achieve through the use of simple dynamic filters in the HTML. Any technique can be used. The important thing to know are the list of vocabs in use for a specific country which are available in the lookups' vocab object as shown below. {{ :rest-api:rest-api-vocab-1.png?nolink |}} ===== Entity Lists ===== The new endpoint to retrieve lists of entities is not currently available in production. Use the old one below for now. This new endpoint will be available soon. Entities such as schools, teachers, students, etc. can be retrieved as a list. Entity Endpoint (GET): https:///api/ Where entity could be: * schools * teachers * students * exams * findata * among others The valid parameters for the query string are the properties of ''[Entity]Filter'' in ''Pineapplies.Data/Models/'' from the [[project_source_code|project source code]] which is public open source. Defaults of values not supplied are provided by the server. They are set in the [Entity]Filter object. Typically these defaults are: * PageNo=1 * PageSize=50 * ColumnSet=0 * SortDirection='asc' * SortColumn= If the user wants all records, not just the first page, they should pass PageSize=0. Some examples of other filtering parameters are included below for some common entities: * schools (District, Island, ElectorateN, Authority) * teachers (Surname, Language, Qualification, District, Authority) * students (StudentGender, StudentEthnicity) query string parameters are not case-sensitive A request to get all the schools might look like this GET /api/schools?PageSize=0 HTTP/1.1 ....(other headers) This returns a JSON object that looks like this containing all the schools. {{ :rest-api:rest-api-entity-list-get-2.png?nolink&400 |}} The list of schools can be "filtered" by query parameters to the URL. For example, retrieving schools with from the District of "Kosrae" you could issue a request such as below. KSA is the code for "Kosrae" which can be retrieved from Lookups. GET https://localhost:44301/api/schools?District=KSA HTTP/1.1 Accept: application/json, text/plain, */* ....(other headers) You would get something like the following containing all schools from Kosrae. {{ :rest-api:rest-api-entity-list-get-1.png?nolink&400 |}} The following is the legacy entity list retrieval REST API endpoint. It should not longer be used. Entities such as schools, teachers, students, etc. can be retrieved as a list. Entity Endpoint (POST): https:///api//collection/filter A request might look like this POST /api/schools/collection/filter HTTP/1.1 ....(other headers) Where entity could be: * schools * teachers * students * exams * findata * among others This returns a JSON object that looks like this. {{ :rest-api:rest-api-list-1.png?nolink&400 |}} The list of schools can be "filtered" by adding some JSON data to the request. For example, retrieving schools with authority of "Baptist Church" you could issue a request such as below. BAP is the code for "Baptist Church which can be retrieved from Lookups. POST https://localhost:44301/api/schools/collection/filter HTTP/1.1 ....(other headers) {"Authority":"BAP","pageNo":1,"pageSize":50,"columnSet":0,"sortColumn":"SchNo","sortDirection":"asc"} ===== Calling the warehouse ===== Warehouse Endpoints all start with the following (though the following in itself does not return anything): https:///api/warehouse While still in development, the basic structure of the endpoints will look like: //[selection]?[report]&.... Broadly, the endpoint will define the contents of the data, while the query parameters will configure how it is presented. The specific parameters support may vary according to the contents. The following endpoints are examples and demonstrate this general format: https:///api/warehouse/enrol/school https:///api/warehouse/enrol/district https:///api/warehouse/enrol/electoratel/ https:///api/warehouse/enrol/district?report https:///api/warehouse/enrol/authority/CDE https:///api/warehouse/enrol/district/CHK?report https:///api/warehouse/enrol/school/CHK001 https:///api/warehouse/enrol/district/PNI?ByAge https:///api/warehouse/enrol/schooltype?byClassLevel=false https:///api/warehouse/flow/school https:///api/warehouse/flow/school/CHK002?asPerc https:///api/warehouse/flow/nation?report https:///api/warehouse/flow/district?report&asPerc ===== Common query parameters ===== "report" indicates "report format". Usually, the data is disaggregated by Gender; i.e. there is a field GenderCode on each row, and the Enrol value is the enrolment for that Gender. Adding the r to the endpoint means that the data does not have separate rows for gender; instead there are field EnrolM, EnrolF, Enrol on each record. This "denormalisation" can be easier to work with sometimes (e.g. in JasperReports,) hence the name "report format". ===== Budgets Endpoints ===== It would be useful to get all the lookups related to budgets data first. https:///api/lookups/collection/findata Then all the following endpoints are available. https:///api/warehouse/finance https:///api/warehouse/finance/nation https:///api/warehouse/finance/district/{filterCode?} (where filterCode is a DistrictCode) https:///api/warehouse/finance/sector/{filterCode?} (where filterCode is a SectorCode) https:///api/warehouse/finance/costcentre/{filterCode?} (NOT YET WORKING) The endpoint above /api/warehouse/finance/costcentre is not yet working ===== Enrol Endpoints ===== Contents 'enrol' returns enrolment data. Aggregation can be by * school * district * electoratel (local electorate) * electoraten (national electorate) * island * region * authority * schooltype * nation Depending on the aggregation above, you can filter for a specific value. In practice, you may prefer just to get all records and filter at the client. 'nation' cannot be filtered. Other configuration are available with option values. By default, enrolment data is grouped by ClassLevel, but not by Age. The following options examples allow you to control this: https:///api/warehouse/enrol/district?byAge=true https:///api/warehouse/enrol/district?byAge=false -- the default https:///api/warehouse/enrol/district?byAge -- same as byAge=true https:///api/warehouse/enrol/district?byClassLevel=true -- the default https:///api/warehouse/enrol/district/?byClassLeve=false -- no disaaggregation by class level https:///api/warehouse/enrol/district?byClassLevel -- sames as byClassLevel=true Using the options, the smallest return set for enrolments you can get is: https:///api/warehouse/enrol/nation?report&byClassLevel=false which returns just one row for each year. The most detailed enrolments data is: https:///api/warehouse/enrol/school?byAge ===== Flow Endpoints ===== 'flow' endpoints return flow rate indicators (e.g. repeat rate, promote rate, dropout rate, survival rate) as well as the data from which these are calculated (i.e. enrolments and repeaters for 2 years.) Flow can be reported by the following aggregation * school * district * nation Flow endpoints also support 'report' option. The rates returned by flow are ratios (i.e. fractional values typically between 0 and 1.) Add the query parameters '?asPerc' to have these returned as percentages. For example, https:///api/warehouse/flow/district/PNI?report&asperc ===== Exams Endpoints ===== 'Exams' endpoints return exams data in a format for flexible analysis. Access it at the following URL. https:///api/warehouse/exams/table And received data that looks like the following. {{ :rest-api:rest-api-exams-1.png?nolink |}} ===== School Accreditation Endpoints ===== All endpoints relating to accreditations are of the form: https:///api/warehouse/accreditations[/][/]?[&report] Where represents the aggregatation applied to the data and can take these values: * school * district * authority * authoritygovt * schooltype * nation * table (the default - aggregation by district, authority, authoritygovt, schooltype) As well as these values, accreditation data is grouped by InspectionResult: Level 1, Level 2, Level 3, Level 4. https:///api/warehouse/accreditations/school https:///api/warehouse/accreditations/district https:///api/warehouse/accreditations/authority https:///api/warehouse/accreditations/schooltype https:///api/warehouse/accreditations/nation https:///api/warehouse/accreditations/table https:///api/warehouse/accreditations GroupValue specifies a particular value of the grouping to filter on. For example: https:///api/warehouse/accreditations/school/PNI001 https:///api/warehouse/accreditations/district/CHK https:///api/warehouse/accreditations/authority/CDE https:///api/warehouse/accreditations/schooltype/K12 Content determines the shape of the returned data set and can take these values: * result: returns the overall inspection result (ie level) of the accreditation. This is calculated as described above. * byStandard: returns a record for each standard, showing the score and calculated result for that standard * performance: return a record for each level of result, showing the number of each criteria and standard within the group that attain that result. (This is based on the TmpPerformance... views that use the legacy accrreditations model) Where 'result' is the default; so that: https:///api/warehouse/accreditations yields the same result as https:///api/warehouse/accreditations/table?result ==== Num vs NumThisYear ==== To measure progress towards accreditation, we need to take snapshots by year of the accreditation status of schools in that year. There are two ways we can look at this: - find the most recent accreditation inspection for each school up to and including the year (i.e. cumulative) - consider and report on only accreditation inspections conducted within the year In any year, it is important to only consider the most recent inspection for each school. Otherwise, results would be distorted by "doubling up" on some schools. Warehouse tables presenting accreditation data use two fields: Num and NumThisYear. * Num is the number of schools meeting the selection criteria of the record, based on the most recent accreditation performed at that school up to and including the 'survey year' of the record. * NumThisYear is the number of schools matching the selection criteria of the record, that have had an accreditation performed in that year. In the web dashboard, charts allow you to present either 'Evaluated In ' (i.e. NumThisYear) or 'Cumulative' (i.e. Num) To illustrate, suppose a school has 2 accreditations: A (Level 2) in 2015, and B (Level 3) in 2017 Then the school will be represented by these accreditations: ^ Year ^ Num ^ NumThisYear ^ | 2014 | - | - | | 2015 | A | A | | 2016 | A | - | | 2017 | B | B | | 2018 | B | - | | 2019 | B | - | Accumulating by result (i.e. total schools in level), we would have //Cumulative:// ^ Year ^ L1 ^ L2 ^ L3 ^ L4 ^ | 2014 | | | | | | 2015 | | 1 | | | | 2016 | | 1 | | | | 2017 | | | 1 | | | 2018 | | | 1 | | | 2019 | | | 1 | | //In Year:// ^ Year ^ L1 ^ L2 ^ L3 ^ L4 ^ | 2014 | | | | | | 2015 | | 1 | | | | 2016 | | | | | | 2017 | | | 1 | | | 2018 | | | | | | 2019 | | | | | Let's continue building on this example, in additon to the school above with 2 accreditations: A (Level 2) in 2015, and B (Level 3) in 2017; we also have another school with 2 more accrediations: C (Level 2) in 2015 and D (Level 4) in 2018. Then the school will be represented by these accreditations: Not sure about the below ^ Year ^ Num ^ NumThisYear ^ | 2014 | - | - | | 2015 | A | A | | 2016 | A | - | | 2017 | B | B | | 2018 | B | - | | 2019 | B | - | Accumulating by result (i.e. total schools in level), we would have //Cumulative:// ^ Year ^ L1 ^ L2 ^ L3 ^ L4 ^ | 2014 | | | | | | 2015 | | 2 | | | | 2016 | | 2 | | | | 2017 | | 1 | 1 | | | 2018 | | | 1 | 1 | | 2019 | | | 1 | 1 | //In Year:// ^ Year ^ L1 ^ L2 ^ L3 ^ L4 ^ | 2014 | | | | | | 2015 | | 2 | | | | 2016 | | | | | | 2017 | | | 1 | | | 2018 | | | | 1 | | 2019 | | | | | ==== Content ==== These options control the format of the data returned by the school accreditations API. * result: tabulates the overall result of the survey. the field InspectionResult holds the result level on each row. * byStandard: Level results are shown by standard. The filed 'stnadrd' appears on each row. Note this view is always is 'report' format, that is, the 8 fields shown above are present on each record. * performance: this view is a crosstabulation designed for specific, pre-exisitng reports. There is a row for each result level, and a column for each standard and criteria. The value for that column is the number of assessments in the group that achived the refenced result value in that criteria or standard. ==== Report Format ==== The option '?report' denormalises the returned data by InspectionResult. That is, instead of the aggregation field InspectionResult, and two data fields (Num, NumThisYear); report format has fields: * Level1 * Level2 * Level3 * Level4 * Level1ThisYear * Level2ThisYear * Level3ThisYear * Level4ThisYear This is a smaller dataset, and convenient for a banded report writer (e.g. Crystal Report, JasperReport.) For 'cube' applications (e.g. pivot tables, crossfilter tableau) use the normalised version of the data previously discussed. Note that report option cannot be used with 'performance' content option. Examples: This section lists illustrative REST calls for schools accreditations. Try these out (e.g. using Fidler, or pacific emis restest) to examine the results. https:///api/warehouse/accreditations -- table view https:///api/warehouse/accreditations/table -- same as above grouping by a particular entity type https:///api/warehouse/accreditations/district https:///api/warehouse/accreditations/schooltype https:///api/warehouse/accreditations/authoritygovt https:///api/warehouse/accreditations/authority https:///api/warehouse/accreditations/school within this, may be filtered to select a single item: https:///api/warehouse/accreditations/district/CHK https:///api/warehouse/accreditations/schooltype/K12 https:///api/warehouse/accreditations/authoritygovt/G https:///api/warehouse/accreditations/authority/COG https:///api/warehouse/accreditations/school/PNI200 National totals: https:///api/warehouse/accreditations/nation Any of these selections can be combined with the 3 possible content selectors: https:///api/warehouse/accreditations/district?byStandard https:///api/warehouse/accreditations/schooltype?byStandard https:///api/warehouse/accreditations/authoritygovt?performance https:///api/warehouse/accreditations/authority?result https:///api/warehouse/accreditations/school --- result is the default As well, you can combine the 'report' option when using byStandard or result content: https:///api/warehouse/accreditations/district?byStandard&report https:///api/warehouse/accreditations/schooltype?byStandard&report https:///api/warehouse/accreditations/authority?result&report https:///api/warehouse/accreditations/school?report ===== WASH Endpoints ===== These endpoints is for WASH data gathered through the Education Survey Tool Official WASH Survey designed in collaboration with UNICEF and fully integrated into the EMIS. Get a list of all the questions on the survey https:///api/warehouse/wash/questions Get all the response for all the schools. https:///api/warehouse/wash You should get data like the following. This includes the schools with their year WASH survey was conducted, their district, authority, authority group, school type and the response to each question. For the Num and NumThisYear data refer to the School Accreditation for explanation. {{ :rest-api:wash-question-answers-1.png?nolink&400 |}} Most of the analysis can be done using those above endpoints. In fact, the Pacific EMIS web app has a single dashboard widgets for those where the user simply selects the question and gets the analysis for that question across all the nation (or with some disaggregation filter applied). However, some of the WASH data is not a simple response but some slightly more involved data. There are currently only two such examples. The endpoints are below and example of the analysis are in the Pacific EMIS WASH dashboard components. https:///api/warehouse/wash/toilets https:///api/warehouse/wash/water ===== Individual Entity Dashboards Endpoints ===== A common place for dashboards are individual entities (e.g. a school, a teacher, a student). This section will provides some details on various API endpoints used in producing those interactive dashboards. ==== An Individual School ==== The following are most of the endpoints needed for an individual school dashboard. /api/warehouse/enrol/school/{schoolID}?report /api/warehouse/enrol/district?report /api/warehouse/enrol/nation?report /api/warehouse/schoolteachercount/{schoolID} (or with ?report if you prefer that data format) /api/warehouse/flow/school/{schoolID}?report&asperc /api/warehouse/exams/school/{schoolID}?report Note that ?report is simply a slightly different format that you may or may not prefer. Check it out. However, those endpoints themselves are not quite enough to produce all the valuable analysis. The above is missing some important details such as school accreditation (or other school inspections) and more details about the school such as its district (for comparison analysis to its own district where it is located.) To produce more useful analysis you might need to retrieve the following which will provide you the school, its district and some school accreditation data for some more important analysis. /api/schools/{schoolID} Note that to access the /api/warehouse you do not need to login. However, to access the above you will need to login with a read only user and provide a bearer token to access the school's endpoint. ===== Future Extensions ===== Other content endpoints will be added/modified for * teachers * schools * leveler (enrolment ratios by education level used for Indicators) * classleveler (enrolment ratios for individual year of education used for Indicators) * budget (high level budget data) * School Accreditation * Exams * WASH ===== Currently Working REST API to be Deprecated ===== Some of the REST API that are currently available and working (and some no longer working) will be deprecated in favor of their final version 1. Anybody still using any of the Deprecate below needs to move to the Instead Use as soon as possible. ==== Used in Exams Dashboard ==== Deprecate https:///api/warehouse/examsdistrictresults Instead use https:///api/warehouse/exams/district/{districtCode?}/r ==== Used in Schools Dashboard ==== Deprecate https:///api/warehouse/tableenrol https:///api/warehouse/tableenrol/r Instead use Unsure yet what to replace this with. It is actually still in use in Pacific EMIS web app and the backend code has no Deprecated attribute like other deprecated end points. Deprecate https:///api/warehouse/districtenrol https:///api/warehouse/districtenrol/r Instead use https:///api/warehouse/enrol/district https:///api/warehouse/enrol/district/r Deprecate https:///api/warehouse/schoolflowrates Instead use https:///api/warehouse/flow/nation ==== Used in Individual School Dashboard ==== Deprecate https:///api/warehouse/enrolbyschool/{schoolNo} Instead use https:///api/warehouse/enrol/school/schoolNo?report Deprecate https:///api/warehouse/schoolflowrates/{schoolNo} Instead use https:///api/warehouse/flow/school/{schoolNo} Deprecate https:///api/warehouse/schoolteachercount/{schoolNo} Instead use https:///api/warehouse/teachers/school/[schoolno]?report ==== Used in Teachers Dashboard ==== Deprecate https:///api/warehouse/teacherpupilratio Instead use https:///api/warehouse/pupilteacherratio Deprecate https:///api/warehouse/schoolteacherpupilratio Instead use https:///api/warehouse/pupilteacherratio/school Deprecate https:///api/warehouse/teachercount Instead use https:///api/warehouse/teachers?report Deprecate https:///api/warehouse/schoolteachercount Instead use https:///api/warehouse/teachers/school?report ==== Used elsewhere ==== Deprecate https:///api/warehouse/v0/edlevelage Instead use https:///api/warehouse/edlevelage ===== Advanced Optimization Usage ===== Users and servers in the Pacific may not have very fast internet connections, so the warehouse gives a few ways to control the size of downloads ==== XML vs JSON ==== The API can return XML or JSON. Is size is an issue JSON might be preferred. The warehouse will give you back XML if you request XML (i.e. if you have text/xml or application/xml in your Accept header:) {{ ::rest-api:rest-api-xml-1.jpg?nolink |}} For JSON, make sure to remove this from Accept and just ask for application/json {{ :rest-api:rest-api-xml-2.jpg?nolink |}} ==== GZip: ==== Add the header Accept-Encoding: gzip to get back gzipped data: Content-Encoding: gzip ==== ETag: ==== Warehouse calls will return an ETag header which indicates the version of the warehouse data. ETag: "5FF37255-5AE8-4A10-9230-A00E15B63C7E" When you call a specific URL for a second time, you can add this ETag as If-None-Match header If-None-Match: "5FF37255-5AE8-4A10-9230-A00E15B63C7E" If the warehouse has not changed (i.e. there has not been a warehouse rebuild since the data was acquired the first time) you will get a 304 - Not Modified response. See how this looks in Fiddler below. {{ :rest-api:rest-api-xml-3.jpg?nolink |}} This gives you the possibility to cache data on the client (e.g. tablet app, web app), together with its ETag, and whenever the app is started, checked whether the data you have is up-to-date. ==== Custom JSON Deflation ==== The warehouse can supply data in a customized format that eliminates the repetition of property names from the JSON collection that is returned. If you add to your warehouse request the header: X-Json-Deflate: ON then you get back a JSON object with these two properties: * columns: an array of strings that is the names of the columns in the data * rows: an array of arrays. Each element in rows is an array of the values of the data in that row. The order is the same as the order of columns. You need to do a bit of postprocessing to turn this back into a standard collection object. For example, here’s what we do in typescript: {{ :rest-api:rest-api-xml-4.jpg?nolink |}} ==== Only get what you need ==== Another way to optimize requested data is to only get what you need. The datasets returned from the warehouse can be aggregated to different levels. If you do not want to report on data at school level, use an end point to get it aggregated to say, District or Authority. Here some comparisons – all results are gzipped Enrolment by school {{ :rest-api:rest-api-xml-5.jpg?nolink |}} Now with X-JSON-Deflate {{ :rest-api:rest-api-xml-6.jpg?nolink |}} And now only getting by District {{ ::rest-api-xml-7.jpg?nolink |}} Furthermore, you can add the query string options byAge and byClassLevel control what data is present in the resultset. {{ :rest-api:rest-api-xml-8.jpg?nolink |}} This set of options is the smallest enrolment dataset you can get back – a mere 215 bytes! There is just one row for each year. These options can be set to true or false (no value = true) {{ :rest-api:rest-api-xml-9.jpg?nolink |}} If the data is Gzipped, Json-Deflate can decrease the size of data between 15%-20%. If not gzipped, Json-Deflate decreases size around 70%.