Differences

This shows you the differences between two versions of the page.

--- emis_developer_manual [2023/07/06 17:42] – [Download and Install the Main Development Tools] ghachey
+++ emis_developer_manual [2026/01/02 20:47] (current) – ghachey
@@ Line 107: / Line 107: @@
   - Download and install [[https://git-scm.com/|Git]]
-  - Download and install latest [[https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx|Visual Studio]] (better tested with 2019).
+  - Download and install [[https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx|Visual Studio]] (better tested with 2019).
   - Download and install [[https://www.microsoft.com/en-us/sql-server/sql-server-downloads|MS SQL Server Developer]] (better tested with 2019; at least one developer uses 2022 without issues).
   - Download and install latest [[https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms|MS SQL Management Studio]]
-  - Download and install most recent version 16 LTS [[https://nodejs.org/en/|NodeJS]].
+  - Download and install most recent version 16 LTS [[https://nodejs.org/en/|NodeJS]]. A reboot may be necessary if you also install all the build tools through the installer.
   - Install Bower from the command line you can simply ''> npm install -g bower''.
   - Install Grunt from the command line you can simply ''> npm install -g grunt-cli''.
   - Download and install [[https://www.sourcetreeapp.com/|SourceTree]] (useful for Git).
+<note important>The latest NodeJS could be made to work with some effort. But last time this was tried there was just a bit more pain then value since this is not part of the application but just the development environment. Using the latest of the version 16 branch works without issues.</note>
 ==== Software Configuration Management ====
@@ Line 177: / Line 177: @@
 {{ :developer-manual:vs-external-tool-config.png }}
-You can then verify if VS is actually using the node and npm binaries you think they were by opening **Tools - NuGet Package Manager - Package Manager Console** and typing the following commands and observing the output.
+You can then verify if VS is actually using the node and npm binaries you think they were by opening **Tools - NuGet Package Manager - Package Manager Console** and typing the following commands and observing the output. The version should be the one you installed manually above (and not the one in the screenshot).
 {{ :developer-manual:vs-confirm-node-npm-version.png }}
@@ Line 1092: / Line 1092: @@
 <note important>One thing that we are not able to deliver from the local server is the google fonts; and the (phasing out) fontawesome fonts. The google fonts have to come from google. So if these are not previously loaded (in browser cache) then we need an internet connection to get them. This is really the only thing that forces us across the internet now. If Pacific EMIS cannot get these google fonts; then the ‘character equivalent’ of each icon in font family material-icons is displayed. Hence you see odd bits of text on the screen where there should be an icon. There is not much we can do about this, but it is worth noting that this is not affected by remote mode; in fact in remote mode, if there is no internet connection, then you would need to get all the libraries from browser cache for the app to have a hope of working.</note>
 ===== Low Level Documentation and API =====
@@ Line 1102: / Line 1103: @@
   * The AtAGlance pages are built from a static XML file containing a table of aggregated values. Particular values are dug out of this XML using Jquery as a kind of client-side "xpath". Some bulk operations in Pineapples pass blocks of XML representing a collection of records into the Sql Server. The Enrolment edit page does this. So the JSON returned from the client is transformed to XML then passed to the SQL server.
-==== Public REST API ====
+==== Notes Teacher Data Processing ====
+=== Collection ===
+  * Census workbook (Excel XLSX)
+  * PDF survey
+In both cases the data are reduced to XML; however the formats are different.
+=== Upload ===
+The Excel and PDF documents are uploaded and processed to extract the XML contents, which are uploaded into the EMIS database.
+=== Main tables ===
+  *   TeacherIdentity: records represent a teacher
+  *   TeacherSurvey: the data collected about each teacher in the annual survey. TeacherSurvey holds the source XML data for each teacher. The formats differ depending on the source (PDF / XLSX). TeacherSurvey is the main operational table of teacher data.
+  *   Also use pTeacherRead.TeacherSurveyV which provides denormalised values (school, surveyYear).
+=== Accessing TeacherSurvey XML ===
+The functions pTeacherRead.Activities and pTeacherRead.TeacherQualCert shred the source XML on TeacherSurvey into relational records.
+=== Main functions ===
+  * Handles both formats (PDF/XLSX) of source XML to produce a common rowset — consumers downstream of these functions deal with a consistent data format regardless of collection source.
+  * Shows the array of Activities — all classes taught, A = Admin and X = Other.
+  * Calculates the "Allocated Class Level" (the most representative single class level for each teacher).
+  * From AllocatedClassLevel, derives Allocated Sector, EdLevel and ISCED.
+  * Uses AllocatedSector and qualification history to determine Qualified and Certified status for each year.
+pTeacherRead.TeacherQualCert has three principal uses:
+  * Provides detailed teacher data to the School page (available without a warehouse rebuild).
+  * Provides the History tab on the Teacher page (no warehouse rebuild required).
+  * Used by the warehouse rebuild; Qualified and Certified status are consolidated in one place.
+=== Warehouse Teacher Info ===
+The fundamental warehouse table for teacher data is warehouse.TeacherLocation. This performs several functions:
+  * For legacy reasons, can incorporate teacher locations sourced from Appointments.
+  * Collects all survey data, Qual/Cert status and allocated Level, Sector and EdLevel from
+pTeacherRead.TeacherQualCert:
+  * Supplies Estimate records for missing surveys.
+  * Includes teaching staff and non‑teaching staff (A, X).
+=== Common filters on TeacherLocation ===
+TAMX in ('T','M') — filters to staff performing some teaching duty. M staff also perform Admin or X Other duties.
+Source not in ('A','A?') — remove any record sourced from Appointment and not supported by survey. Note that TAMX in ('T','M') excludes these records as well.
+Use the Source filter to get all survey teacher info, including Estimates and Admin/Other staff.
+Grouping by AllocatedEdLevel, AllocatedSector, AllocatedSCED one can then COUNT(*) or SUM(Qualified), SUM(Certified).
+Note: Aggregates on warehouse.TeacherLocation vs pTeacherRead.TeacherQualCert differ because TeacherLocation provides the Estimate records. These queries return the same record count:
+<code sql>
+Select * from pTeacherRead.TeacherQualCert(null,null,null,2024, null,0)
+Select * from warehouse.TeacherLocation WHERE Source not in ('A','A?') And Estimate = 0
+</code>
+All teacher reporting from the warehouse should ultimately draw from warehouse.TeacherLocation so that interpretations of Allocated and Qualified/Certified match those embodied in pTeacherRead.TeacherQualCert.
+=== Examples ===
+<code sql>
+Select * from warehouse.TeacherLocation WHERE Source not in ('A','A?') — all staff derived from surveys (including Estimates from previous surveys).
+Select * from warehouse.TeacherLocation WHERE TAMX in ('T','M') — all staff with at least one teaching activity.
+Select * from warehouse.TeacherLocation WHERE TAMX = 'A' — all staff exclusively admin workers.
+Select * from warehouse.TeacherLocation WHERE A = 1 — staff doing some admin duties, not necessarily exclusively (i.e., may be teaching as well).
+</code>
+There is one record per teacher per year in TeacherLocation (i.e., tID/surveyYear is unique). Use TeacherLocation for lists of teachers as well as aggregates.
+warehouse.TeacherLocation includes Gender and Age Group; for other demographics, including name, join back to TeacherIdentity.
+=== Weightings ===
+A major group of tables derived from warehouse.TeacherLocation are:
+  * warehouse.TeacherActivityWeights
+  * warehouse.TeacherActivityWeightsEdLevel
+  * warehouse.TeacherActivityWeightsSector
+  * warehouse.TeacherActivityWeightsClassLevel
+To handle the fact that a teacher may perform multiple activities, weightings are applied to teacher activities in the aggregation category to avoid double‑counting when aggregating.
+Example (UIS A9 sheet): we record the number of teachers teaching in ISCED 1 and the number teaching in ISCED 2, then add totals across the page. If a teacher teaches one class at ISCED 1 and another at ISCED 2, counting them as 1 in each would inflate the page total.
+To overcome this, we weight a teacher's activities as a fraction of all their teaching activities. If Teacher A has 2 activities (1 in ISCED 1 and 1 in ISCED 2), their weighting for ISCED 1 is 0.5 and for ISCED 2 is 0.5, so their contribution to the page total is 1.
+=== Terminology ===
+FTPT (Full Time / Part Time) weighting — teaching activities in some group (ISCED, EdLevel, Sector, Class Level) as a fraction of the total teaching activities of that teacher.
+FTE (Full Time Equivalent) — teaching activities in some group as a fraction of the TOTAL activities of that teacher (including any A or X activity).
+Example:
+Teacher B teaches 1 class at ISCED 1 and is recorded as performing Admin tasks. FTPT for ISCED 1 = 1 (1/1 teaching activity). FTE = 0.5 (1 divided by 2 total activities).
+Difference between FTPT and FTE depends on whether the teacher performs A or X activities. Aggregated, the difference indicates teacher time spent on admin tasks.
+Filter by TAMX in ('T','M') to consider only teacher activities, consistent with UIS Survey page A9.
+A third weighting mechanism is Allocation: allocate a teacher to a single AllocatedClassLevel, and from this to a single AllocatedISCED, AllocatedEDLevel, AllocatedSector. This allocation is produced in pTeacherRead.Activities and is used in the weighting tables.
+Example:
+Teacher C teaches 3 classes at ISCED 1 and 1 at ISCED 2. AllocatedISCED = ISCED 1 (most activity). FTPT for ISCED 1 = 0.75, for ISCED 2 = 0.25. Using Allocation: Allocated ISCED 1 = 1, Allocated ISCED 2 = 0. Page total remains 1.
+TeacherActivityWeights entries for Teacher C:
+ISCED 1A — W (FTE): 0.75, WTeach (FTPT): 0.75, Allocated: 1
+ISCED 2A — W: 0.25, WTeach: 0.25, Allocated: 0
+Summary: a teacher will have entries in this table (for a given year) for each ISCED Level in which they teach; exactly one of these records will have Allocated = 1.
+=== Head Count (unweighted totals) ===
+Head Count = number of teachers who do any activity in that group regardless of other activities. One teacher may be included in multiple categories; head counts cannot be summed without double counting.
+To get Head Count by EdLevel:
+<code sql>
+select surveyYear, EdLevelCode, count(tID) HeadCount
+from warehouse.TeacherActivityWeightsEdLevel
+where TAMX in ('T','M')
+group by surveyYear, EdLevelCode
+</code>
+To get a Head Count grand total (without double-counting):
+<code sql>
+select surveyYear, count(DISTINCT tID) HeadCount
+from warehouse.TeacherActivityWeightsEdLevel
+where TAMX in ('T','M')
+group by surveyYear
+</sql>
+=== Handling Admin and Other (A / X) teachers ===
+A/X staff have no teaching class level activities; their allocated Class Level (and hence Allocated ISCED, ED Level, Sector) is derived entirely from the default level of the school type.
+A/X activities are written to separate rows, identified by the flag AX: null for teaching activities, or 'A' or 'X'.
+To split these off from teaching activities when aggregating:
+<code sql>
+select surveyYear, coalesce(AX, EdLevelCode) EdLevelCode, count(tID) HeadCount
+from warehouse.TeacherActivityWeightsEdLevel
+group by surveyYear, coalesce(AX, EdLevelCode)
+</code>
+Contributions of staff with TAMX in ('A','X'), as well as A/X activities for teachers with TAMX = 'M', are now grouped separately.
+=== VERMPAF ===
+Values from the Weights tables are incorporated into the VERMPAF Teachers node. These are filtered on TAMX in ('T','M') and are consistent with aggregates on the Weights tables when those filters are applied. This filter also makes these totals (for FTE and FTPT) compatible with the UIS Survey.
-Refer to [[emis_public_rest_api|EMIS Public REST API]]
 ==== A Note on the use of Lodash's Memoize Function in Dashboard Components  ====