Differences Between 2.1 and 2.0.2 TVStudy 2.1 is the result of many months of work. This work was undertaken to fully implement the TV IX Check mode, convert the software to function entirely in the NAD83 datum, and provide new features and improved performance. During this process, much of the code in TVStudy was examined, edited, reorganized, and/or corrected along the way. As such, this changelog lists all major changes/updates but does not name every individual issue which has been addressed since the 2.0.2 release. Interference (IX) Check The TV Interference (IX) Check mode, initially released as a beta version in 2.0 through 2.0.2 for trial testing, has been extensively modified and now more closely matches the processes used by the older tv_process software. The TV IX Check study setup dialog has been removed as all of the settings can now be managed in other locations in the software. Every TV IX Check study outputs a file named “tvixstudy.dat” which mimics the output of tv_process. Studies in this mode conducted with TVStudy 2.0.2 or older should be discarded and recreated in TVStudy 2.1. In the TV IX Check mode, it is now possible to name a study when run from the Java UI. When running a TV IX Check study from the GUI and replicating the study record, you can now put multiple channel numbers in the "Rep. Channel" box using commas and dashes (e.g., “15,16,18-22"). TVStudy will automatically queue up one study run for each channel. To keep the studies from causing duplicate name errors, TVStudy adds " (on NN)" to the end of the study name, where NN is the replication channel. When creating a series of TV IX Check studies for multiple replicated channels, the study record's own channel may be included in the channel list even for a digital study record. That will cause the wizard to automatically set up a non-replicated study for the record, in addition to one or more replicated studies. This will be done only when more than one channel appears in the replication channel field; entering the same channel as a single channel input for a digital record results in an error. Entering the same channel in any manner for an analog study record is not an error; that will create a study for on-channel replication to digital service. To get a non-replicated study for an analog record the replicate option must not be checked. TV IX Check mode supports the 1% contour increase and 5% population decrease tests that will be used during the first 90 days after the Incentive Auction Closing and Channel Reassignment Public Notice is issued. Additionally, to facilitate post-auction filings, TVStudy has the ability via a study parameter to exclude records filed before a specific date unless there is an associated baseline record on that channel. This will allow applications to be analyzed with respect to post-auction records. There is no longer a setting to select the station data used for the AM station search in TV IX Check mode. If no suitable data set is found, the study run will note in the report output that the AM check could not be performed, then proceed with the rest of the study. (See below for more information on the AM check.) The default study template for TV IX Check mode can be set by naming a suitable template “Interference Check”. Similarly, the default output file settings can be set by naming those settings “Interference Check”. It is also possible to set a default set of station data in the same manor, but it is more likely that you will want to use the “Most recent TV” data option, which is the default if a station dataset named “Interference Check” does not exist. Added a filing cut-off date field to the wizard dialog and web servlet form for setting up TV IX Check studies. Entering a date in this field will cause records with a sequence date on or after the cut-off date to be excluded from the study. If the study record is an LPTV/translator, the exclusion applies only to other LPTV/translator records. If the study record is full-service or Class A, the exclusion applies to all records. Note that this feature is not a "roll-back" to a previous point in time. In other words, it does not deal with records that are archived when the study is run but were not archived as of the cut-off date. TV6<->FM Implemented TV6<->FM interference study mode. The options for FM into DTV 6 interference are to use 73.525 directly with extrapolation to DTV service levels, or use a fixed D/U table derived from the 2007 NPR report. 1 Geographies/Recv Pats TVStudy can now perform analyses on individual fixed location points. These points are defined using the new “Geography Editor” that can be used for various alternate study analyses, not just points mode. Additionally, points can be analyzed using alternate user-provided receive antenna patterns. A receive pattern can have different azimuth patterns for a set of frequencies, edited in a manner similar to a matrix elevation pattern. Both geographies and receive antenna patterns can be imported and exported. Alternate Service Areas Added the ability to project contours by various new methods, including contours based on Longley-Rice projections, or FCC contours with a fixed distance or percentage added. In the study parameters, a default contour mode can now be set which applies to all desired stations in the study that have the default contour service area mode. New parameters to select different methods of determining study point location(s) in each cell have been added. These are population-weighted centroid of Census points in cell (existing method), geographic center of the cell, largest Census point in cell, and all Census points in cell (each point gets actual population and a fraction of cell area based on total population in the cell). There is a second new parameter to choose whether or not to use the actual Census point location that is nearest to the previously-determined location. Changed the undesired record search in the study engine to use the actual service area boundary instead of the "rule extra distance" approximation. However, the check of the actual boundary vs. the extra distance with a warning if the extra distance is too small still occurs, since the GUI application still 1 NPR Labs. (2007). Interference Rejection Thresholds of Consumer Digital Television Receivers on Channel 6 with FM Broadcast Signals. Retrieved from http://www.nprlabs.org/sites/nprlabs/files/documents/reports/20071217C%20Report.pdf and it was filed on the record in docket RM-11579. uses the extra distance algorithm for scenario-building searches. If the warning appears from the study engine, some undesireds may not have been included in the scenario. Datasets All coordinates in all station datasets are now stored in the NAD83 datum. For LMS datasets, this means TVStudy has moved from using the NAD27 fields to the NAD83 fields. For CDBS datasets, this means that TVStudy converts all of the coordinates when the dataset is imported or, for datasets already in TVStudy during the database update, at update time. A consequence of this change is that new XML exports now use different attribute names for coordinates, LATITUDE83 and LONGITUDE83, which are unambiguously NAD83. The previous attribute names LATITUDE and LONGITUDE are generally assumed to be NAD27 coordinates during import, except for wireless records which have always been NAD83. For the sake of clarity, XML import now recognizes attributes LATITUDE27/LONGITUDE27 to explicitly provide coordinates in NAD27. "Most recent" options now appear in menus for selecting station data for a record search or study build. These options automatically select an appropriate imported data set with the most recent data. The test is based on the content of the import, not when it was imported. Most-recent items are provided for LMS TV, CDBS TV, and CDBS FM, plus a "Most recent TV" option that will select either LMS or CDBS. Wireless data sets do not contain date information so this feature is not available for those. Imported data sets are no longer considered "in use," allowing a data set to be deleted at any time. Existing studies built from a now-deleted data set can still be used and further modified using data from any currently-available data set. However, in such a study it may not be possible to revert a record that has been made editable because the original data set is no longer available. During LMS record translation, the city and state are now retrieved from fields in the application_facility and lkp_dtv_allotment tables, rather than the facility table. However, call signs are still retrieved from the facility table for consistency with CDBS. CDBS translation continues to use the facility table for all three pieces of information. Call sign searches will now match regardless of leading D characters appearing in the database fields, e.g. a search for WXYZ will also match DWXYZ. However, a search for DWXYZ will not match WXYZ. A new record status has been added for experimental facility records. The status EXP is assigned when the facility status field contains "EXPER". The EXP status is assigned regardless of the underlying APP/CP/LIC/STA status; that underlying status is no longer available. Fixed an issue where the optional name for an imported data set was not being checked for uniqueness during import. This is now checked; however, the import does not fail if the name is not unique, the name is just not saved. The name can be changed later as needed. TVStudy now displays CDBS-style antenna ID number for LMS records instead of a UUID string. AM station data tables in LMS are now supported. The table files will be read during an LMS data set import if possible, however the files are not required. If the AM files are missing, the import will continue but AM data will not be available in that data set. The AM station search done during an TV IX Check study using an LMS data set will now attempt to query the LMS AM tables. If those tables are not present, e.g. the files were not present at import, or if any error occurs during the queries, the AM check will not be performed but the study will otherwise continue. Note this feature is only partially tested because the AM files do not yet appear in LMS downloads. Sample files were used to simulate the presence of the data for testing. However, if the actual files that eventually appear in the LMS downloads do not have the same structure as the samples, future LMS imports with AM data could fail entirely. If the files are present, they must be in the expected format; apparent errors within the files are not ignored. New "generic" data set import capability has been added. The concept here is that a generic data set is created empty, then once it exists you can import things into that set from any format supported (currently just XML). Then you can add more by importing more things in the same or different formats. In other words, the data set is not just a single fixed import, you can build a whole database by importing pieces from different sources. This is a work in progress and is expected to be updated in a future release. Wizards/Windows Expanded the UI for creating types of studies that pre-select a target record (i.e. TV IX Check and TV6<- >FM) to allow the selected record to be replicated to a different channel before starting the study. Changed the logic for assigning study names to studies created with the "wizard" dialogs, e.g. TV IX Check. The name entered in the dialog no longer has to be unique. When the study is actually created, if the name already exists, a numerical suffix e.g. "(123)" will be appended to make the new name unique. The separate run windows have been replaced with a single “Run Manager” window which contains all of the runs that are active or completed and not yet removed. Study-build dialogs now have two action buttons, “Build” and “Build & Run.” The Build button will cause the study build to occur (in the run manager as usual) but without a subsequent run of the study. Build & Run builds and then runs the study as in previous versions. Addressed editing issues in the scenario and record editors in TV IX Check, wireless -> TV, and TV6<->FM studies. To prevent the scenario from inadvertently being made invalid, the channel and geographic location of the study record now cannot be changed even if the record is otherwise editable. This includes not allowing the record to be replicated, or if it is already a replication, not allowing it to be reverted to the original channel. Also, for a DTS record, individual DTS sites cannot be added or removed. Output TVStudy now sets the output file selection in an output settings window. It is possible to save preferred configurations of output files and map files and choose them at run-time or while configuring the study. The output file options and the map file options are on separate tabs. The scenario name is now written in the KML output file element instead of just the file name. TV IX Check study runs now have the option to only output detail files as requested for scenarios that fail the interference check. Restored antenna ID value to parameters CSV file, and when replicating by equal-area method, the antenna ID from the original record is preserved. Created composite coverage output as a new option in the output file settings for examining combined coverage across multiple desired station records. When selected, there will be a composite coverage total in both the summary and detail report files, and if map files are requested, a composite coverage points file is created called "cmpcvpts". The point coordinates in the composite map files are always the cell center point, not the actual study point in the cell. If selected, the composite map output will be generated using the cell center point regardless of the mode used to define study points in cells. If the "all Census points" mode is used, all points in a given cell will be considered individually when accumulating the final result for that cell. That means if just one of those study points has coverage, the composite output will indicate the entire cell has coverage, which limits its usefulness and is not recommended. Note that the points output files ("points" in the map files and the "points.csv" file) benefit from the new code added to support composite mode. Regardless of whether composite mode is active, the new code for determining the coverage status across the entire cell grid for a scenario is used to support the points output. As a result, those outputs will no longer contain duplicate points. New map output settings have been added. Provides options for additional columns in the coverpts file, as well as general options like excluding points with no service or no population. Each option for output in coverpts may activate multiple actual columns, such as separate latitude and longitude columns if the study point coordinates are selected. TVStudy can now output maps as images, but the option only appears if TVStudy detects an available version of the GhostScript software installed. The user needs to install GhostScript, then manually create a symbolic link in TVStudy’s lib/ directory to the actual "gs" command wherever it is located. Note that image maps will only be generated from a scenario that has exactly one desired station. Changes have been made to attribute options and behaviors for map output files. Additional attributes are now included in both the points map file and points CSV file, including ground elevation, receiver height (in points study mode), and land-cover and clutter types (when clutter adjustments are being applied). When the output coordinates are at the cell center in the points file, the study point coordinates will automatically be included in the file as attributes. Changed the report and CSV labels to "Service Area" instead of "Noise-Limited", since that can now be something other than a noise-limited contour. If the record is being studied in unrestricted mode, those columns are blank or say "(unrestricted)" with no numbers. Individual Census point information can now be included in the detail cell file. If enabled, following each "P" record describing a study point is a list of new "C" records each describing one Census point used in defining that study point. The "C" records include the Census block ID (a unique identifier in each country's Census data that can be used to match other Census data attributes), the population, and the household count for each Census point. Note the Census point information is currently not saved in the study engine cache files. As a result, if a detail cell file is requested, even when study points and field- strength calculation results are reloaded from cache, the study engine must still repeat the Census data search queries to reconstruct the point lists. This significantly reduces the performance benefit of caching; however, those repeat queries will occur only when a detail cell file is requested and the option is enabled. Site number for desired DTS transmitters are now included in cell file outputs, following the desired source key. The column is always present, but is empty if the record is not a DTS desired. Additionally, the site number for each DRT is now included in the sources section of the cell file outputs. Fixed the rules summary printout to show FM study rules correctly. Study Editor, Parameters, and Templates Fixed problems throughout the study setup logic where channel range was not being checked, incorrectly allowing some records on out-of-range channels to be added to a study. Added a "Rename..." item to the Template menu in the Template Manager. This allows templates to be renamed even after they are locked. Added NLCD 2011 data as an option for clutter adjustments. Major improvements have been made to the antenna pattern editor, allowing matrix patterns to be fully edited including converting a non-matrix pattern to a matrix (or vice-versa), adding and removing pattern slices, and changing the azimuths of existing pattern slices. Fixed bugs in the scenario editor. When the filtering menus at the bottom of the window were used, the filter applied to the visible content of the table; however, the filter was not applied when a menu item or button was used. The requested action would thus be performed on a different record than appeared to be selected. Fixed bug in elevation pattern handling that caused the pattern mirroring and built-in electrical tilt detection features to work incorrectly when the elevation pattern maximum was not at 0 degrees. Added an option to derive a horizontal-plane pattern for contour projection if a record has a real elevation pattern and mechanical tilt, or has a matrix pattern. Also added a new option in the output file settings to write horizontal plane patterns to CSV files. Added an option in the "Add Many" scenario-building dialog to search the baseline record table. When baseline is selected, search options for service and country are available but selecting anything other than US full-service DTV will yield an empty result in current datasets. The service and country criteria are enabled because future LMS imports may contain baseline records in other countries or with other service codes. TVStudy now supports facility ID values less than or equal to 0. Facility ID values in most situations can now have any value. Two exceptions are in a pair study baseline scenario and in the CDBS/LMS TV baseline record tables, facility IDs must still be greater than 0. Changed the logic for retrieving call sign, city, and state from CDBS/LMS, including FM. First try the facility table, if the field is blank there, fall back to the application or application_facility table. In the study editor "Study" tab, there is a new menu to pick the propagation model used when running that study. Currently the options are "Longley-Rice", "FCC curves", and "Free space". New models can be added, as described in the TVStudy source files (look in the "src/model/" directory at the files "model.h" and "model.c"; comments in those files will guide you from there). When a new model is added in those code files, it will automatically appear in the UI. The Java application queries the study engine for a list of available models when it starts up, so it is necessary to restart the Java UI to load new models. When a new DTS record is created from a non-DTS record, the non-DTS record is used to set the DTS reference facility as well as the DTS "parent" properties. Previously when that was done, the reference facility was changed to digital. Now it is left alone, which means you can create a DTS record with an analog reference. An editable DTS record can now have a sectors boundary area. The parameter for mirroring on generic patterns has been expanded from a simple yes-no option to a menu with choices "Never", "Always", and "Propagation only." Changes have been made to the XML import code to improve performance and to fix a number of bugs. When records need to be retrieved from a data set by record ID search, rather than doing lookups one record at a time while parsing the XML, record IDs are accumulated then all retrieved in a single batch query at the end. Imports in which a large number of records need to be retrieved, especially from LMS data, are now dramatically faster. Added better reporting messages following an XML import during which no errors occurred but nothing was successfully imported. The message will indicate if the XML file did not contain a recognized structure, or if all records were ignored due to being an incompatible type (such as FM when importing into a TV study). In the scenario editor, it is now possible to include the same record more than once as long as each appearance is replicated to a different channel. Fixed an issue causing the Site # tabs in the source editor for a DTS station to be out of numerical order. The single-record search UI layout has been re-arranged (the "Add One" and the study-build "wizard" windows). Mostly this is a cosmetic change and all the UI elements still function as before, but there is a new option to search by country, and country is now shown in the record table. The study and template editor UI layouts have been changed. Parameters are in their own tab, and parameter sections are picked from a drop-down menu rather than tabs. In the study editor, the "Study" tab still exists to show those settings that can't be handled as parameters. The "Add Many" search dialog in the scenario editor now has the ability to save search criteria for later re-use. When importing station data from XML, up to two station data sets can now be selected for resolving record elements that reference a data set record by ID. This allows, for example, selecting one LMS and one CDBS data set so either type of ID can be resolved, when the content of the XML being imported is mixed or unknown. Otherwise import behavior is unchanged; if a by-reference element cannot be resolved it is not imported but the rest of the import continues. All file open and save dialogs are now configured to require an appropriate file name extension; the "All Files" choice is no longer available in most cases. For example, when importing XML data, only files with names ending in ".xml" can be chosen. Likewise, on XML export, the file created will always end in ".xml" regardless of whether or not that is actually entered in the file name field. Default parameter values for the rule-extra-distance ERP table have been modified slightly. The ERP for the low-to-mid ERP break is changed from 0.25 kW to 0.2 kW. Note this change applies only to the default template. Existing studies and templates will not be affected by the change; it will only take effect for templates created after the update. Added a new option parameter to disable the rule-extra-distance ERP table and instead use the maximum desired signal calculation distance as the rule-extra-distance for all searches. This is meant to be used with alternate service area methods, or altered service contour levels, that result in service areas substantially larger than standard FCC contours. Particularly, this option is needed to avoid rule- extra-distance failures when using the unrestricted service area mode. Note this option should never be set in a template used to create a TV IX Check study. Doing so would result in frequent study build failures because the additional stations appearing in undesired searches cause the scenario-generation tree to become unmanageably large. Changed the behavior of TV contour distance limit parameters. These now apply to all contours including Longley-Rice, and they apply when the contour is projected rather than as an overlay during point analysis, so the contour actually appears truncated in map outputs. The new maximum desired signal distance parameter is checked in all cases including contours and geographies. Points that fall inside a service area but beyond the maximum distance will still appear in outputs but with a -999 field strength indicating no calculation actually occurred. Command line utility TVStudy 2.1 includes a command line utility called “ptelev” that makes some of the TVStudy internal functionality available from the command line. It reads a configuration file at "lib/ptelev.conf" and otherwise is run with the “lib/ptelev” command. See page Error! Bookmark not defined. for more information on this utility. TVStudy 2.1 also provides a simpler command line utility called “nadcon” that allows for coordinate conversions only using a minimum number of files. No support is provided for this utility. Underlying Data Status feedback is now provided during long database updates, such as the one for TVStudy 2.1. Added a households field to all population tables. Household values and tallies appear in all CSV and cell output files immediately following population. Population data has been re-processed to add household counts and will be reimported when TVStudy is updated. After a new database installation or after some major upgrades (including TVStudy 2.1), TVStudy will search for XML files located in the xml/ directory and attempt to import those as templates or geographies. Optimized the terrain cache memory management algorithm for improved behavior. In the previous version, memory use could become excessive under some conditions, leading to memory compression and virtual memory swapping. Out-of-memory errors in environments with no swap space should now be much less likely. Improved the caching algorithm used for land-cover database lookups. The previous algorithm was seriously deficient once the cache reached the "full" state, causing very long run-times for large-area studies with clutter adjustment applied. The new algorithm is much better, although there is still a 20- 30% performance penalty when applying clutter adjustments in a large-area study (previously that was 300-500%). TVStudy 2.1 comes with a utility to convert 1/3-second terrain for use in TVStudy. This feature is currently unsupported. Other Added DTS self-interference analysis, activated with new parameter, and added supporting parameters. Self-interference will appear in detail report and CSV files as a separate "undesired" line. In the DTS self- interference analysis, all signals outside time arrival limits are RSS to a composite undesired signal rather than being checked individually for D/U failure. DTS self-interference analysis is output to the detail cell file using new record formats S and DS. Improvements have been made to the handling of insufficient-memory situations. If the study engine does abort for that reason, the error message is now visible in the run window. However, that should never occur now because the UI asks the engine for a report on memory at startup and if there isn't enough, the UI won't even let study runs be started at all. You can continue editing in the UI as much as you like, but you can't run anything. Fixed a bug where a study point result code could incorrectly be the 11, 12, or 13 value indicating a propagation model error when the result was not in doubt due to the error. Under some conditions, the error flag could be set due to an undesired-signal calculation on a point that did not have service due to desired signal below threshold, with no error on the desired-signal calculation. In that situation the undesired-signal error is irrelevant and should not affect the result status. Fixed a bug where changes to the rule-extra-distance parameters did not always take immediate effect. Sometimes the actual distances used in database searches would not update until the study was closed and re-opened. Fixed a bug that could cause incorrect use of invalid cached calculation results after certain types of concurrent edits (changes by two different applications sharing the same database installation).