Differences

This shows you the differences between two versions of the page.

--- emis_developer_manual [2023/02/20 21:49] – [Download and Install the Main Development Tools] ghachey
+++ emis_developer_manual [2026/01/02 20:47] (current) – ghachey
@@ Line 63: / Line 63: @@
 There is no need for an expensive license, the Express edition which is freely available will work fine for most small countries. If you don't have already a commercial license download the free version [[https://www.microsoft.com/en-us/download/details.aspx?id=29062|MS SQL Server Express]]. Any of the files would have the required minimal database engine but might has well download the 64bit full version called //ENU\x64\SQLEXPRADV_x64_ENU.exe//. Double click and follow through the installation. Choosing the default values is a good start and reboot the system when done.
+The following databases distributed with the source code in the folder databases:
+  * **IdentifiesP.bak**: a database that handles the users, permissions and customization of the navigation menus.
+  * **pacificemisdb.bak**: the primary database containing all the tables, views, stored procedures and functions but with no data at all.
+  * **pacificemisdb-withsamplelookups.bak**: the primary database containing all the tables, views, stored procedures and functions with some sample data particularly common lookup data and meta data for basic configuration.
+<note important>The Pacific EMIS is based on a fairly advanced and complex database. It typically requires a fair amount of the knowledge about the inner workings of the system to correctly configure it for specific needs of a country. While the above database will provide a running system (with no data) it would be unlikely ready for deployment into a country without further customization typically done directly in the data and also via our custom per country template technique.</note>
+Restoring a database from a backup is documented [[https://msdn.microsoft.com/en-us/library/ms177429.aspx|here]]. Take note of the names of your new databases and also the name of the server (e.g. LOCALHOST, HOSTNAME\SQLEXPRESS) you will need those next.
 ==== Programming Language ====
@@ Line 97: / Line 107: @@
   - Download and install [[https://git-scm.com/|Git]]
-  - Download and install latest [[https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx|Visual Studio]] (better tested with 2019).
+  - Download and install [[https://www.visualstudio.com/en-us/products/visual-studio-community-vs.aspx|Visual Studio]] (better tested with 2019).
-  - Download and install [[https://www.microsoft.com/en-us/sql-server/sql-server-downloads|MS SQL Server Developer]] (currently moving to 2019).
+  - Download and install [[https://www.microsoft.com/en-us/sql-server/sql-server-downloads|MS SQL Server Developer]] (better tested with 2019; at least one developer uses 2022 without issues).
   - Download and install latest [[https://docs.microsoft.com/en-us/sql/ssms/download-sql-server-management-studio-ssms|MS SQL Management Studio]]
-  - Download and install most recent version 16 LTS [[https://nodejs.org/en/|NodeJS]].
+  - Download and install most recent version 16 LTS [[https://nodejs.org/en/|NodeJS]]. A reboot may be necessary if you also install all the build tools through the installer.
   - Install Bower from the command line you can simply ''> npm install -g bower''.
   - Install Grunt from the command line you can simply ''> npm install -g grunt-cli''.
   - Download and install [[https://www.sourcetreeapp.com/|SourceTree]] (useful for Git).
+<note important>The latest NodeJS could be made to work with some effort. But last time this was tried there was just a bit more pain then value since this is not part of the application but just the development environment. Using the latest of the version 16 branch works without issues.</note>
 ==== Software Configuration Management ====
@@ Line 146: / Line 156: @@
 <note important>When doing the bower install in the Pineapples.Client project you will be faced with a JQuery conflict resolution. It is actually important to select 2.2.4 otherwise the dashboards will be broken. Refer to https://bitbucket.org/softwords/pineapples/issues/1147/many-dashboard-broken-in-latest-build for details</note>
-Then install a location version of grunt and all the dependencies.
+Then a local version of grunt seems to be needed to silence an error in Visual Studio later. Again, from the command line go into the ``pineapples`` root folder (not ``Pineapples.Client``) and issue the following command.
+<code>
+> npm install grunt
+</code>
+Then install a locale version of grunt and all the dependencies in the ``Pineapples.Client``.
 <code>
@@ Line 161: / Line 177: @@
 {{ :developer-manual:vs-external-tool-config.png }}
-You can then verify if VS is actually using the node and npm binaries you think they were by opening **Tools - NuGet Package Manager - Package Manager Console** and typing the following commands and observing the output.
+You can then verify if VS is actually using the node and npm binaries you think they were by opening **Tools - NuGet Package Manager - Package Manager Console** and typing the following commands and observing the output. The version should be the one you installed manually above (and not the one in the screenshot).
 {{ :developer-manual:vs-confirm-node-npm-version.png }}
@@ Line 167: / Line 183: @@
 </note>
-Try to build the solution. At first, this will automatically get VS to start downloading some NuGet dependencies so will take longer the first time around. Once the build works you can try a _build_deploy of the frontend.
+As a one time thing, you need to set/verify the starting project of the solution. Right click on the **Pineapples** project in the **Solution Explorer** and click on **Set as StartUp project**. The ``Pineapples`` project should from now on become bold.
+At this point if you followed closely the previous sections you should already be able to right click on the **Solution 'pineapples'** and click **Build Solution**. The first time around this will require the downloading of all the NuGet packages (npm and bower dependencies would have already been pulled, at least on a good working Internet connection). It should build 4 projects without errors.
+You should then try running the grunt task _build_deploy as shown below. Run it a couple of times for good measure.
+{{ :developer-manual:grunt-build-deploy.png }}
+If you do not yet see the Task Runner Explorer window you can bring it up by right clicking on the file ``Pineapples.Client -> gruntfile.js`` and then clicking on ``Task Runner Explorer`` as shown below.
+{{ :developer-manual:task-runner-explorer.png?nolink |}}
+==== Application Configuration ====
+The main configuration for the application is the file ''Web.config'' from the ''Pineaples'' project. There are a few things that need to be configured before you can run the application. The relevant parts of the ''Web.config'' configuration file are detailed here.
+You will need to setup two databases: the Identity database (i.e. IdentitiesP) and the application's main database (e.g. MIEMIS, SIEMIS, SIEMIS-TEST, KIEMIS). You should have those already in your MS SQL Server as detailed in previous section.
+Replace ''localhost'' with your server name and SQL server instance name (e.g. MACHINENAME, MACHINENAME\SQLEXPRESS). The appSettings's database needs to be the one you want to work with (e.g. MIEMIS-TEST). The context is not the same as the database but would be the place app name (e.g. miemis, kemis, siemis, fedemis)
+<code xml>
+<add name="DefaultConnection" connectionString="Data Source=localhost;Initial Catalog=IdentitiesP;...
+</code>
+<code xml>
+<appSettings>
+  <add key="server" value="localhost" />
+  <add key="database" value="siemis" />
+  <add key="context" value="siemis" />
+  <add key="title" value="FedEMIS Online" />
+  ...
+</appSettings>
+</code>
+There are other appSettings less critical but important for some functionalities.
+<code xml>
+<appSettings>
+  ...
+  <!-- The report server configuration for the reporting functionality -->
+  <add key="ReportServerUrl" value="http://localhost:8080/reportserver" />
+  <add key="jasperUrl" value="http://data.pss.edu.mh:8080/jasperserver" />
+  <add key="jasperUser" value="jasperadmin" />
+  <add key="jasperPass" value="Adm1n2p$$16!" />
+  ...
+  <!-- The location where files are stored on the computer used by a number of features in the system -->
+  <add key="FileDb" value="C:\files\miemis\filedb" />
+  ...
+  <!-- Emailing functionality cnofiguration -->
+  <add key="emailSenderName" value="EMIS Web Administrator" />
+  <add key="emailSenderAddress" value="no-reply@national.doe.fm" />
+  <!-- use the your own email if you want to test receiving systems emails (e.g. errors notifications) -->
+  <add key="emailReceiverAddress" value="ghachey@nuzusys.com" />
+  <!-- use the bc for testing, so that a copy of all outbound emails is sent to this address -->
+  <!--<add key="emailBcc" value="edemisaudit@softwords.com.au" />-->
+  ...
+</appSettings>
+</code>
+And finally, for email notifications to work it must be configured to use a email server. Various approaches can be used, perhaps the easiest is to use a Gmail account with an application password.
+<code xml>
+  <system.net>
+    <mailSettings>
+      <!-- settings for email delivery using smtp client
+      will vary according to the sender - refer to sysadmin docs at
+      http://docs.pacific-emis.org/doku.php?id=systems_administrator_manual#setting_up_email-->
+      <!-- Using a Google account is an easy and reliable way.
+           this may require that you "Access for less secure apps"
+           be enabled for the Gmail account used by the NetworkCredential
+           see https://www.google.com/settings/u/1/security/lesssecureapps -->
+      <smtp deliveryMethod="Network">
+        <network host="smtp.gmail.com" port="587"
+                 userName="youremail@gmail.com" password="yourapppassword"
+                 defaultCredentials="false" enableSsl="true" />
+      </smtp>
+    </mailSettings>
+  </system.net>
+</code>
+==== Running the Solution ====
+At this point you should be ready to run the system by clicking on the following. Note that below was first set to Google Chrome but it could be started within any browser.
+{{ :developer-manual:vs-run-project.png }}
-TODO
 ==== [Optional] Various Visual Studio Configuration ====
@@ Line 212: / Line 312: @@
   * **Tabify Selected Lines** - converts leading spaces to tabs. Respects the tab size setting in **Tools - Options** when making this conversion. It is recommended to run this on files before submitting a commit.
   * **View White Space** - toggle display spaces and tabs (CTRL-E,S)
-==== Databases Setup ====
-You will need to setup two databases: the application's main database (e.g. MIEMIS, SIEMIS, SIEMIS-TEST, KIEMIS) and the Identify database (i.e. IdentitiesP). Those should be given to you by one of the developers with appropriate knowledge and authority until a sample database and created and available for use by all stored in database folder.
-Restoring a database from a backup is documented [[https://msdn.microsoft.com/en-us/library/ms177429.aspx|here]]. Take note of the names of your new databases and also the name of the server (e.g. LOCALHOST, HOSTNAME\SQLEXPRESS) you will need those next.
-In VS, open the file ''Web.config'' from the ''Pineaples'' project. Two places you need to configure. Locate the following line.
-<code xml>
-<add name="DefaultConnection" connectionString="Data Source=localhost;Initial Catalog=IdentitiesP;...
-</code>
-And replace ''localhost'' with your server name (e.g. MACHINENAME\SQLSERVER2012). Then locate the following lines.
-<code xml>
-<appSettings>
-  <add key="server" value="localhost" />
-  <add key="database" value="siemis" />
-  <add key="context" value="siemis" />
-  <add key="title" value="FedEMIS Online" />
-  <add key="ReportServerUrl" value="http://localhost:8080/reportserver" />
-</appSettings>
-</code>
-And replace ''localhost'' with your own server name (e.g. same as above MACHINENAME\SQLSERVER2012) and ''siemis'' with the name of your database (e.g. fedemis-test). The context is not the same as the database but would be the place app name (e.g. miemis, kemis, siemis, fedemis)
-==== Building the Solution ====
-First, as a one time thing, you need to set the starting project of the solution. Right click on the **Pineapples** project in the **Solution Explorer** and click on **Set as StartUp project**. The Pineapples project should from now on become bold.
-At this point if you followed closely the previous sections you should already be able to right click on the **Solution 'pineapples'** and click **Build Solution**. The first time around this will require the downloading of all the NuGet packages (npm and bower dependencies would have already been pulled, at least on a good working Internet connection). It should build 4 projects without errors.
-You should then try running the grunt task _build_deploy as shown below. Run it a couple of times for good measure.
-{{ :developer-manual:grunt-build-deploy.png }}
-At this point you should be ready to run the system by clicking on the following. Note that below was first set to Google Chrome but it could be started within any browser.
-{{ :developer-manual:vs-run-project.png }}
 ==== Troubleshooting Visual Studio Build/Run Errors ====
@@ Line 1035: / Line 1092: @@
 <note important>One thing that we are not able to deliver from the local server is the google fonts; and the (phasing out) fontawesome fonts. The google fonts have to come from google. So if these are not previously loaded (in browser cache) then we need an internet connection to get them. This is really the only thing that forces us across the internet now. If Pacific EMIS cannot get these google fonts; then the ‘character equivalent’ of each icon in font family material-icons is displayed. Hence you see odd bits of text on the screen where there should be an icon. There is not much we can do about this, but it is worth noting that this is not affected by remote mode; in fact in remote mode, if there is no internet connection, then you would need to get all the libraries from browser cache for the app to have a hope of working.</note>
 ===== Low Level Documentation and API =====
@@ Line 1045: / Line 1103: @@
   * The AtAGlance pages are built from a static XML file containing a table of aggregated values. Particular values are dug out of this XML using Jquery as a kind of client-side "xpath". Some bulk operations in Pineapples pass blocks of XML representing a collection of records into the Sql Server. The Enrolment edit page does this. So the JSON returned from the client is transformed to XML then passed to the SQL server.
-==== Public REST API ====
+==== Notes Teacher Data Processing ====
+=== Collection ===
+  * Census workbook (Excel XLSX)
+  * PDF survey
+In both cases the data are reduced to XML; however the formats are different.
+=== Upload ===
+The Excel and PDF documents are uploaded and processed to extract the XML contents, which are uploaded into the EMIS database.
+=== Main tables ===
+  *   TeacherIdentity: records represent a teacher
+  *   TeacherSurvey: the data collected about each teacher in the annual survey. TeacherSurvey holds the source XML data for each teacher. The formats differ depending on the source (PDF / XLSX). TeacherSurvey is the main operational table of teacher data.
+  *   Also use pTeacherRead.TeacherSurveyV which provides denormalised values (school, surveyYear).
+=== Accessing TeacherSurvey XML ===
+The functions pTeacherRead.Activities and pTeacherRead.TeacherQualCert shred the source XML on TeacherSurvey into relational records.
+=== Main functions ===
+  * Handles both formats (PDF/XLSX) of source XML to produce a common rowset — consumers downstream of these functions deal with a consistent data format regardless of collection source.
+  * Shows the array of Activities — all classes taught, A = Admin and X = Other.
+  * Calculates the "Allocated Class Level" (the most representative single class level for each teacher).
+  * From AllocatedClassLevel, derives Allocated Sector, EdLevel and ISCED.
+  * Uses AllocatedSector and qualification history to determine Qualified and Certified status for each year.
+pTeacherRead.TeacherQualCert has three principal uses:
+  * Provides detailed teacher data to the School page (available without a warehouse rebuild).
+  * Provides the History tab on the Teacher page (no warehouse rebuild required).
+  * Used by the warehouse rebuild; Qualified and Certified status are consolidated in one place.
+=== Warehouse Teacher Info ===
+The fundamental warehouse table for teacher data is warehouse.TeacherLocation. This performs several functions:
+  * For legacy reasons, can incorporate teacher locations sourced from Appointments.
+  * Collects all survey data, Qual/Cert status and allocated Level, Sector and EdLevel from
+pTeacherRead.TeacherQualCert:
+  * Supplies Estimate records for missing surveys.
+  * Includes teaching staff and non‑teaching staff (A, X).
+=== Common filters on TeacherLocation ===
+TAMX in ('T','M') — filters to staff performing some teaching duty. M staff also perform Admin or X Other duties.
+Source not in ('A','A?') — remove any record sourced from Appointment and not supported by survey. Note that TAMX in ('T','M') excludes these records as well.
+Use the Source filter to get all survey teacher info, including Estimates and Admin/Other staff.
+Grouping by AllocatedEdLevel, AllocatedSector, AllocatedSCED one can then COUNT(*) or SUM(Qualified), SUM(Certified).
+Note: Aggregates on warehouse.TeacherLocation vs pTeacherRead.TeacherQualCert differ because TeacherLocation provides the Estimate records. These queries return the same record count:
+<code sql>
+Select * from pTeacherRead.TeacherQualCert(null,null,null,2024, null,0)
+Select * from warehouse.TeacherLocation WHERE Source not in ('A','A?') And Estimate = 0
+</code>
+All teacher reporting from the warehouse should ultimately draw from warehouse.TeacherLocation so that interpretations of Allocated and Qualified/Certified match those embodied in pTeacherRead.TeacherQualCert.
+=== Examples ===
+<code sql>
+Select * from warehouse.TeacherLocation WHERE Source not in ('A','A?') — all staff derived from surveys (including Estimates from previous surveys).
+Select * from warehouse.TeacherLocation WHERE TAMX in ('T','M') — all staff with at least one teaching activity.
+Select * from warehouse.TeacherLocation WHERE TAMX = 'A' — all staff exclusively admin workers.
+Select * from warehouse.TeacherLocation WHERE A = 1 — staff doing some admin duties, not necessarily exclusively (i.e., may be teaching as well).
+</code>
+There is one record per teacher per year in TeacherLocation (i.e., tID/surveyYear is unique). Use TeacherLocation for lists of teachers as well as aggregates.
+warehouse.TeacherLocation includes Gender and Age Group; for other demographics, including name, join back to TeacherIdentity.
+=== Weightings ===
+A major group of tables derived from warehouse.TeacherLocation are:
+  * warehouse.TeacherActivityWeights
+  * warehouse.TeacherActivityWeightsEdLevel
+  * warehouse.TeacherActivityWeightsSector
+  * warehouse.TeacherActivityWeightsClassLevel
+To handle the fact that a teacher may perform multiple activities, weightings are applied to teacher activities in the aggregation category to avoid double‑counting when aggregating.
+Example (UIS A9 sheet): we record the number of teachers teaching in ISCED 1 and the number teaching in ISCED 2, then add totals across the page. If a teacher teaches one class at ISCED 1 and another at ISCED 2, counting them as 1 in each would inflate the page total.
+To overcome this, we weight a teacher's activities as a fraction of all their teaching activities. If Teacher A has 2 activities (1 in ISCED 1 and 1 in ISCED 2), their weighting for ISCED 1 is 0.5 and for ISCED 2 is 0.5, so their contribution to the page total is 1.
+=== Terminology ===
+FTPT (Full Time / Part Time) weighting — teaching activities in some group (ISCED, EdLevel, Sector, Class Level) as a fraction of the total teaching activities of that teacher.
+FTE (Full Time Equivalent) — teaching activities in some group as a fraction of the TOTAL activities of that teacher (including any A or X activity).
+Example:
+Teacher B teaches 1 class at ISCED 1 and is recorded as performing Admin tasks. FTPT for ISCED 1 = 1 (1/1 teaching activity). FTE = 0.5 (1 divided by 2 total activities).
+Difference between FTPT and FTE depends on whether the teacher performs A or X activities. Aggregated, the difference indicates teacher time spent on admin tasks.
+Filter by TAMX in ('T','M') to consider only teacher activities, consistent with UIS Survey page A9.
+A third weighting mechanism is Allocation: allocate a teacher to a single AllocatedClassLevel, and from this to a single AllocatedISCED, AllocatedEDLevel, AllocatedSector. This allocation is produced in pTeacherRead.Activities and is used in the weighting tables.
+Example:
+Teacher C teaches 3 classes at ISCED 1 and 1 at ISCED 2. AllocatedISCED = ISCED 1 (most activity). FTPT for ISCED 1 = 0.75, for ISCED 2 = 0.25. Using Allocation: Allocated ISCED 1 = 1, Allocated ISCED 2 = 0. Page total remains 1.
+TeacherActivityWeights entries for Teacher C:
+ISCED 1A — W (FTE): 0.75, WTeach (FTPT): 0.75, Allocated: 1
+ISCED 2A — W: 0.25, WTeach: 0.25, Allocated: 0
+Summary: a teacher will have entries in this table (for a given year) for each ISCED Level in which they teach; exactly one of these records will have Allocated = 1.
+=== Head Count (unweighted totals) ===
+Head Count = number of teachers who do any activity in that group regardless of other activities. One teacher may be included in multiple categories; head counts cannot be summed without double counting.
+To get Head Count by EdLevel:
+<code sql>
+select surveyYear, EdLevelCode, count(tID) HeadCount
+from warehouse.TeacherActivityWeightsEdLevel
+where TAMX in ('T','M')
+group by surveyYear, EdLevelCode
+</code>
+To get a Head Count grand total (without double-counting):
+<code sql>
+select surveyYear, count(DISTINCT tID) HeadCount
+from warehouse.TeacherActivityWeightsEdLevel
+where TAMX in ('T','M')
+group by surveyYear
+</sql>
+=== Handling Admin and Other (A / X) teachers ===
+A/X staff have no teaching class level activities; their allocated Class Level (and hence Allocated ISCED, ED Level, Sector) is derived entirely from the default level of the school type.
+A/X activities are written to separate rows, identified by the flag AX: null for teaching activities, or 'A' or 'X'.
+To split these off from teaching activities when aggregating:
+<code sql>
+select surveyYear, coalesce(AX, EdLevelCode) EdLevelCode, count(tID) HeadCount
+from warehouse.TeacherActivityWeightsEdLevel
+group by surveyYear, coalesce(AX, EdLevelCode)
+</code>
+Contributions of staff with TAMX in ('A','X'), as well as A/X activities for teachers with TAMX = 'M', are now grouped separately.
+=== VERMPAF ===
+Values from the Weights tables are incorporated into the VERMPAF Teachers node. These are filtered on TAMX in ('T','M') and are consistent with aggregates on the Weights tables when those filters are applied. This filter also makes these totals (for FTE and FTPT) compatible with the UIS Survey.
-Refer to [[emis_public_rest_api|EMIS Public REST API]]
 ==== A Note on the use of Lodash's Memoize Function in Dashboard Components  ====