Skip to main content

Virtual Machine Functions

SAGE Script​

The SAGE Script (shelter-sage-app) monitors SAGE export directories for file changes. When detected, it parses the new or updated sales order CSV files, processes the data into a structured format, and sends it to the main Shelter web application via an API call.

For more details on the SAGE application, creating exports, and field definitions, see the SAGE Documentation.

Application Workflow​

  1. Initialization: The script loads its configuration from settings.json and parses the current and historical sales order CSV files on startup, sending any existing data to the server.
  2. File Monitoring: It continuously monitors both the active and historical sales order CSV files by periodically computing their hashes. It looks for active orders every 5 minutes and historical orders every 15 minutes.
  3. Parsing: When a file change is detected and the hash stabilizes, the script parses the CSV file using the expected headers defined in settings.json.
  4. Data Processing: The flat CSV data is transformed into a structured dictionary, grouping items under their respective sales order numbers along with customer and order information.
  5. Transmission: The data is chunked into manageable batches (configured by row_count, defaulting to 200, though previously around 1000) and sent to the Shelter web application via HTTP POST requests in JSON format.

Configuration​

A settings.json file is required in the same directory as the script. Key configuration options include:

  • site_url: The base URL of the Shelter web application.
  • api_key: Authorization bearer token for the API.
  • controller: The API controller path (e.g., api).
  • function: The specific function endpoint (e.g., import_orders or read_sales_data).
  • file_path: Directory path where SAGE exports the CSV files.
  • sales_file_name: Name of the active sales order CSV file.
  • historical_file_name: Name of the historical sales order CSV file.
  • row_count: The maximum number of records to send in a single API request batch.
  • sales_headers & historical_headers: Arrays defining the expected CSV column names.
  • escape_item_codes: Array of specific item codes to ignore when parsing.

Python Dependencies​

  • csv: Parses the SAGE export CSV files.
  • hashlib: Computes file hashes to detect when the CSV is modified.
  • json: Converts data for API transmission and handles local output storage.
  • logging: Records interactions, errors, and system status to local log files and standard output.
  • requests: Sends POST requests to the Shelter web application.
  • time / datetime: Handles delays and timestamp formatting.
  • threading: Allows simultaneous monitoring of both active and historical CSV files.

(Note: The system historically or in other components may also utilize watchdog for file changes, socket for connectivity checks, and os for file permission adjustments).

Shelter Application Integration​

The Python script communicates directly with the App\Controllers\Api\Sage controller in the Shelter web application. The endpoints used in the script's configuration are mapped to the import_orders method in this controller.

  • import_orders: Accepts batches of sales order records. It validates the data format and structure depending on whether it is historical or active data. The SageDataImporter class is then utilized to either create new sales order records (Sales_order_model) or update existing ones, resolving any data conflicts.
  • Job Management: The system automatically creates related factory jobs (such as EIFS, Pre-Puff, Sill Seal, Pool Steps, Lines, and Shape Mold) based on the product_line of the items in the order.
  • Auto-completion & Deletion: When an order is marked as completed (C) by SAGE, the system automatically finishes associated factory jobs (e.g., ships Sill Seal boxes, completes EIFS targets). If an order is marked as deleted (X), it automatically removes the associated jobs from the application to keep the factory floor clear.

Data Transmission & Storage​

Transmitted Payload: The Python script sends JSON data in batches. The payload structure is:

{
  "is_historical": 0,
  "data": {
    "123456": {
      "order_info": {
        "status": "O",
        "type": "S",
        "order_date": "10/24/2023"
      },
      "customer_info": {
        "customer_po_number": "PO-999",
        "billing_address": {
          "name": "Customer Name",
          "street1": "123 Billing St",
          "street2": "",
          "city": "Cityville",
          "state": "NY",
          "zipcode": "12345"
        },
        "shipping_address": {
          "name": "Customer Name",
          "street1": "123 Shipping St",
          "street2": "",
          "city": "Cityville",
          "state": "NY",
          "zipcode": "12345"
        }
      },
      "items": [
        {
          "item_code": "EF1",
          "item_code_description": "EIFS Block",
          "item_type": "1",
          "unit_of_measurement": "EACH",
          "quantity": "50",
          "unit_price": "10.00",
          "unit_cost": "5.00",
          "extension_amount": "500.00",
          "product_line": "EIFS"
        }
      ]
    }
  }
}

Storage Discrepancies:

  • The transmitted JSON dictionary groups all rows by their sales_order_number.
  • Instead of just saving the data verbatim, the server parses the items array to generate distinct job tickets across different specialized database tables (like Eifs_job_model, Prepuff_job_model, etc.) based on the product_line logic mapping.
  • Cost attributes and completion dates are continually reconciled against the ERP data, so the Shelter application's database mirrors the active truth of the SAGE server.

Temperature Script​

The Temperature Script (temp_sensor) pulls data from the temperature sensor via FTP and sends it to the main Shelter web application. The script operates in a continuous loop to monitor temperature readings and alarms.

Application Workflow​

  1. Load Settings: The script reads its configuration parameters from a local settings.json file.
  2. Retrieve Data via FTP: An FTP command script (ftp_commands.txt) is dynamically generated to log into the FTP server and download the current day's temperature and alarm log files. The script executes the FTP commands using the OS command line.
  3. Parse Files: The script reads the downloaded files (Temperature Trend_Center_{date}.txt and Alarm_{date}.txt), which are encoded in UTF-16, parsing the tab-delimited data and correctly formatting timestamps.
  4. Send Data: The parsed temperature readings and alarm logs are chunked into smaller groups based on the configured batch_size and sent to the Shelter web application via HTTP POST requests to the appropriate API endpoints.
  5. Wait: The script sleeps for a configured duration (read_interval in seconds) before restarting the loop.

Configuration​

A settings.json file is required in the same directory as the script. It must contain the following keys:

  • site_url: The base URL of the Shelter web application.
  • api_key: Authorization bearer token for the API.
  • controller: The API controller path.
  • temp_function: The API endpoint for posting temperature data.
  • alarm_function: The API endpoint for posting alarm data.
  • ftp_host: Hostname or IP of the FTP server.
  • ftp_user: FTP username.
  • ftp_password: FTP password.
  • download_path: The local directory where FTP files should be downloaded.
  • read_interval: The delay (in seconds) between iterations.
  • batch_size: The maximum number of records to send in a single API request.

Python Dependencies​

  • csv: Parses the downloaded tab-delimited log files.
  • time / datetime: Handles delays and formats timestamps.
  • json: Converts data for API transmission.
  • logging: Records interactions, errors, and system status to local log files and standard output.
  • requests: Sends POST requests to the Shelter web application.
  • codecs: Handles UTF-16 encoding when reading files.
  • os / sys: Executes the FTP commands and manages console output.

Shelter Application Integration​

The Python script communicates directly with the App\Controllers\Api\Temperature controller in the Shelter web application. The endpoints used in the script's configuration are mapped to specific methods in this controller:

  • import_temps: Accepts batches of temperature readings. It validates the data (PSEUDOCENTEMP and Date), checks if a record already exists for the timestamp to prevent duplicates, and either updates or creates new entries using the Temperature_model.
  • import_alarms: Accepts batches of alarm events. It validates the alarm Date and Message. For each alarm, it ensures no duplicate notification exists. If it is a new alarm, the controller finds the temperature reading stored in the database that is chronologically closest to the alarm time, appends it to the alarm message, and saves it as a "Temp Alarm" notification using the Notification_model.

Data Transmission & Storage​

Transmitted Payload: The Python script sends JSON data in batches. The payload structure is:

{
  "temperatures": [
    {
      "Date": "YYYY-MM-DD HH:MM:SS",
      "PSEUDOCENTEMP": "72.5"
    }
  ],
  "alarms": [
    {
      "Date": "YYYY-MM-DD HH:MM:SS",
      "Message": "Alarm description"
    }
  ]
}

Storage Discrepancies:

  • The PSEUDOCENTEMP key is renamed and stored as temp in the Temperature_model database table.
  • Alarm data is not stored in a dedicated alarms table. Instead, it is stored generically in the Notification_model table.
  • The Message transmitted from the alarm is modified before storage; the server appends the closest chronological temperature reading to the end of the text (e.g., Alarm description<br><br>Current Temperature: 72.5).

Scales Script​

[!WARNING] DEPRECATED: The Scale Script is currently deprecated due to a recent Merger & Acquisition (M&A). Shelter Enterprises is in the process of transitioning to use the newly acquired company's API. The documentation below is retained for legacy purposes.

The Scale Script (shelter-ent-scale-app) is a Python application that reads weights directly from a connected serial scale and transmits the data to the Shelter web application. It features a full-screen GUI designed to run on a Raspberry Pi to provide visual feedback to operators.

Application Workflow​

  1. Initialization: The script loads configuration variables from config.json and attempts to connect to the physical scale via a USB serial port.
  2. GUI Display: A full-screen Tkinter interface is launched to display the current weight and bag number to the operator.
  3. Continuous Monitoring: The script operates in an infinite loop, continuously polling the scale via serial communication every 3 seconds.
  4. Data Processing: Upon receiving a valid, non-negative weight reading, it parses the string for weight and unit. It will automatically convert pound/ounce (lboz) formats to purely pounds (lb).
  5. Transmission: A JSON payload containing the station, weight, unit, and scale configuration is sent via HTTP POST to the Shelter application.
  6. Feedback: The server responds with either a bag number (success) or an error. The Tkinter GUI updates its display color and text based on the response.

Configuration​

A config.json file is required in the same directory. Keys include:

  • site_url: The base URL of the Shelter web application.
  • api_key: Authorization bearer token.
  • station: Identifier for the physical station where the scale is located (e.g., "Downstairs").
  • scale: Numeric identifier for the scale.
  • controller: The API controller path (e.g., api).
  • function: The specific function endpoint (e.g., scan_bag).

Python Dependencies​

  • pyserial: Handles USB serial communication with the scale hardware.
  • numpy: Used for array manipulation during weight string parsing.
  • requests: Sends POST requests to the Shelter web application.
  • json: Parses the API response.
  • Tkinter: Generates the full-screen visual GUI for operators.
  • logging: Records interactions to standard output and rotating daily log files.
  • socket: Performs basic connectivity checks (e.g., pinging Google).

Data Transmission & Storage​

Transmitted Payload: The Python script sends the following flat data via an HTTP POST request:

{
  "station": "Downstairs",
  "weight": 105.5,
  "unit": "lb",
  "scale": "1"
}

Storage Discrepancies:

  • The transmitted data is processed by the server (e.g., via the scan_bag function) to log a finished bag for an ongoing job at the specified station.
  • The server does not merely store this payload verbatim; it creates a bag record in the database related to an active job.
  • The server responds with the newly generated bag_num identifier (e.g., {"bag_num": 123}), which is not part of the transmitted payload but is displayed on the local Raspberry Pi screen. If no job is active, the server returns an error.

Pre-Expander Script​

The Pre-Expander Script (shelter-preexp-app) automatically detects changes in CSV files exported by the pre-expander machinery, processes the data, and sends it to the main Shelter web application.

Application Workflow​

  1. Initialization: The script loads its configuration from settings.json and parses the current month's CSV file (e.g., YYYY_MM_PJX.csv) on startup, sending any existing data to the server.
  2. File Monitoring: It continuously monitors the CSV file for modifications by periodically computing the file hash (every sleep_time seconds).
  3. Parsing: When a change is detected and the file hash stabilizes, it reads the new rows. It parses the data using predefined headers mapping out silo activity, timestamps, and lot/bag numbers.
  4. Data Formatting: The script groups the flat CSV data by silo_num into a structured JSON payload.
  5. Transmission: The JSON is sent via HTTP POST to the Shelter application.
  6. Graceful Restart: The script checks the time and automatically exits at 11:59 PM to allow log rotation and clear memory. When paired with a Windows Scheduled Task, this ensures daily stability.

Configuration​

The settings.json file controls the script behavior with the following keys:

  • site_url: The base URL of the Shelter web application.
  • file_path: Directory path where the machinery exports the CSV files.
  • data_folder: Directory for storing local output files (e.g., JSON dumps).
  • api_key: Authorization bearer token for the API.
  • controller: The API controller path (e.g., api/preexpand).
  • function: The specific function endpoint (e.g., import_data).
  • sleep_time: Interval (in seconds) between file change checks.
  • headers: An array of strings defining the expected CSV column names.

Python Dependencies​

  • csv: Parses the tab-delimited pre-expander files.
  • hashlib: Computes file hashes to detect when the CSV is modified.
  • json: Structures the data payload for transmission.
  • logging: Manages rotating local log files and console output.
  • requests: Sends POST requests to the Shelter web application.
  • time / datetime: Handles polling intervals and time-based checks.

Shelter Application Integration​

The Python script communicates directly with the App\Controllers\Api\Preexpand controller in the Shelter web application, predominantly utilizing the import_data endpoint:

  • import_data: Accepts a grouped dictionary of silos and their corresponding expansions. It performs robust validation (including handling edge cases like "No full" end times). For each expansion, it calls a helper to match against an existing pre-expansion job using a unique key, or creates a new job record in the Preexp_job_model. It then processes the materials used by updating Lot_model records to reflect that specific bags of raw materials were consumed during the expansion. The endpoint returns an updated snapshot of available inventory back to the script.

Data Transmission & Storage​

Transmitted Payload: The Python script sends JSON data grouped by silo numbers. The payload structure is:

{
  "silos": {
    "1": {
      "expansions": [
        {
          "date": "MM/DD/YYYY",
          "time": "HH:MM:SS AM",
          "silo_start": "MM/DD/YYYY HH:MM:SS AM",
          "silo_end": "MM/DD/YYYY HH:MM:SS AM",
          "code": "A123",
          "lots": [
            { "lot": "LOT_NAME", "bag": "123" },
            { "lot": "LOT_NAME", "bag": "124" },
            { "lot": "LOT_NAME", "bag": "125" }
          ]
        }
      ]
    }
  }
}

Storage Discrepancies:

  • The flat silo_num string identifier (e.g., "1") is converted to an actual database relation (silo_id) referencing the Silo_model.
  • The date and time values sent in the payload are ignored by the server. Only silo_start and silo_end are utilized (stored as start_time and end_time in the Preexp_job_model).
  • The lots array is not stored verbatim. Instead, the server finds the corresponding lot entries in the inventory (Lot_model) and links their database IDs to the job via bag_id_1, bag_id_2, and bag_id_3 columns.
  • The server generates its own internal key_string to uniquely identify each job, which is not part of the transmitted payload.