Printer Friendly, PDF & Email Printer Friendly, PDF & Email

§170.315(g)(3) Safety-enhanced design

Updated on 03-11-2024
Regulation Text
Regulation Text

§ 170.315 (g)(3) Safety-enhanced design—

  1. User-centered design processes must be applied to each capability technology includes that is specified in the following certification criteria: paragraphs (a)(1) through (5), (9) until the criterion's expiration date, and (14), (b)(2), (3) and (11) of this section.
  2. Number of test participants. A minimum of 10 test participants must be used for the testing of each capability identified in paragraph (g)(3)(i) of this section.
  3. One of the following must be submitted on the user-centered design processed used:
    1. Name, description and citation (URL and/or publication citation) for an industry or federal government standard.
    2. Name the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing user-centered design standards was impractical.
  4. The following information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied:
    1. Name and product version; date and location of the test; test environment; description of the intended users; and total number of participants;
    2. Description of participants, including: sex; age; education; occupation/role; professional experience; computer experience; and product experience;
    3. Description of the user tasks that were tested and association of each task to corresponding certification criteria;
    4. The specific metrics captured during the testing of each user task performed in (g)(3)(iv)(C) of this section, which must include: task success (%); task failures (%); task standard deviations (%); task performance time; and user satisfaction rating (based on a scale with 1 as very difficult and 5 as very easy) or an alternative acceptable user satisfaction measure;
    5. Test results for each task using the metrics identified above in paragraph (g)(3)(iv)(D) of this section; and
    6. Results and data analysis narrative, including: major test finding; effectiveness; efficiency; satisfaction; and areas for improvement.
  5. Submit test scenarios used in summative usability testing.
Standard(s) Referenced
Certification Dependencies

Design and Performance: This certification criterion was adopted at § 170.315(g)(3), and is required for all developers seeking certification to § 170.315(a)(1) through (5), a(9) until the criterion's expiration date, (a)(14), (b)(2), (b)(3) or (b)(11) of this section. 

Revision History
Version # Description of Change Version Date
1.0

Initial publication

03-11-2024

This Test Procedure illustrates the test steps required to certify a Health IT Module to this criterion. Please consult the most recent ONC Final Rule on the Certification Regulations page for a detailed description of the certification criterion with which these testing steps are associated. ONC also encourages developers to consult the Certification Companion Guide in tandem with the test procedure as it provides clarifications that may be useful for product development and testing.

Note: The tests step order does not necessarily prescribe the order in which the tests should take place.

Testing components

Documentation Icon Visual Inspection Icon No Test Tool Icon No ONC Supplied Test Data Icon No SVAP Icon

The ONC Health IT Certification Program requires that a limited set of quantitative data elements related to the safety-enhanced design (SED) testing be reported and displayed to the public via the new Certified Health IT Product List (CHPL). Reporting instructions to the CHPL, including data elements and their definitions, formats, and allowable values, may be found at CHPL SED Guide.

System Under Test Test Lab Verification

The health IT developer submits documentation demonstrating that user-centered design (UCD) process(es) were applied to all of the safety-enhanced design referenced certification criteria for which these corresponding capabilities are being presented for certification.

The tester verifies the submitted documentation outlines user-centered design (UCD) process(es) to each capability for any of the following safety-enhanced design criteria submitted for certification:

§ 170.315 (a)(1)   Computerized Provider Order Entry (CPOE) – medications
§ 170.315 (a)(2)   CPOE – laboratory
§ 170.315 (a)(3)   CPOE – diagnostic imaging
§ 170.315 (a)(4)   Drug-drug, drug-allergy interaction checks for CPOE
§ 170.315 (a)(5)   Demographics
§ 170.315 (a)(9)   Clinical decision support (expires on January 1, 2025)
§ 170.315 (a)(14) Implantable device list
§ 170.315 (b)(2)   Clinical information reconciliation and incorporation
§ 170.315 (b)(3)   Electronic prescribing
§ 170.315 (b)(11) Decision support interventions 


System Under Test Test Lab Verification

The Heath IT Module must have conducted summative usability testing with a minimum of ten test participants for each safety-enhanced design criterion capabilities for which certification is being sought.

The tester verifies at least ten test participants, representative of the intended user population, participated in summative usability testing for each safety-enhanced design criterion and the associated capabilities.


System Under Test Test Lab Verification

The health IT developer submits documentation outlining the user-centered design (UCD) process(es) used for each of the safety-enhanced design criteria (specified in section (g)(3)(i)) submitted for testing that includes one of the following:

  1. Where one of the industry or federal government standards is used:
    • Name;
    • Description;
    • Citation (URL and/or publication citation) reference of the UCD industry standard (e.g ISO 9241-210, ISO 13407, ISO 16982, ISO/IEC 62366, and NISTIR 7741);

OR
 

  1. Where an industry or federal government standard is NOT used:
    • Name and citation (URL and/or publication citation) of the industry standard process(es) that formed the basis of the “custom” process(es);
    • Outline of the process(es);
    • Short description of the process(es) used;
    • Explanation of the reason(s) why use of any of the existing UCD standards was impractical.

The tester verifies user-centered design (UCD) process(es) have been documented for each of the safety-enhanced design either by:

  1. Where one of the industry or federal government standards is used:
  • Name;
  • Description;
  • Citation (URL and/or publication citation) reference of the UCD industry standard (e.g ISO 9241-210, ISO 13407, ISO 16982, ISO/IEC 62366, and NISTIR 7741);

OR
 

  1. Where an industry or federal government standard is NOT used:
  • Name and citation (URL and/or publication citation) of the industry standard process(es) that formed the basis of the “custom” process(es); 
  • Outline of the process(es);
  • Short description of the process(es) used;
  • Explanation of the reason(s) why use of any of the existing UCD industry or federal government standards was impractical.

System Under Test Test Lab Verification

The health IT developer must have conducted summative usability testing and provides the required documentation for each of the safety-enhanced design criteria (specified in section (g)(3)(i)) submitted for summative usability testing, using:

  • The NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing content report for usability test report(s) that address the application of the documented and referenced UCD process(es)

The health IT developer may provide the required information/sections from NISTIR 7742 in any format, provided that the required information is included.

The tester verifies the existence and adequacy of the summative usability test report(s) for each safety-enhanced design criterion (specified in (g)(3)(i)) capability being presented for testing/certification. The tester verifies the report(s) conform(s) to the content and completion requirements of the required information/sections of NISTIR 7742. The tester verifies that the name and version of the product are the final version (release) of the product for which the health IT developer is seeking certification.


System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • Date and location of the test
  • Test environment
  • Description of the intended users
  • Total number of participants

The tester verifies the existence and adequacy of the required NISTIR 7742 information/sections.


System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • Description of participants (i.e., Sex, Age, Education, Occupation/role, Professional experience, Computer experience, Product experience)

The tester verifies the demographic characteristics of the subject pool meet the specifications of the particular requirement (NISTIR 7742 3.1 “Participants”); where use conditions and population of the users are analyzed to ensure participant characteristics reflect the audience of current and future users.


System Under Test Test Lab Verification
  1. The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:
    • Description of the user tasks (task scenarios) that were tested and association of each task to corresponding certification criteria
  1. The tester verifies the user tasks employed in the study are prioritized in accordance with the risk associated with user errors (NISTIR 7742 3.3 “Tasks”).
  2. The tester verifies that the test scenarios included in the NISTIR 7742 content report for each of the UCD Required Criteria are inclusive of the tasks or functionality the health IT developer provided for testing to the certification criterion.

System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • The specific metrics captured during the summative testing of each user task performed in (g)(3)(iv)(C) of this section (Task Success (%), Task Failures (%), Task Standard Deviations of Task Performance Time (%), Task Performance Time, User Satisfaction Rating (Scale with 1 as very difficult and 5 as very easy) or an alternative acceptable user satisfaction measure.)

The tester verifies the specified metrics are captured in the report.


System Under Test Test Lab Verification

The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:

  • Test results for each task using metrics listed above in (g)(3)(iv)(D).

The tester verifies test results are provided for each task using the specified metrics in (g)(3)(iv)(D).


System Under Test Test Lab Verification
  1. The summative usability testing report includes NISTIR 7742 Customized Common Industry Format (CIF) Template for Electronic Health Record Usability Testing information/sections:
    • Results and data analysis narrative (i.e., Major test finding, Effectiveness, Efficiency, Satisfaction, Areas for improvement). Measures of satisfaction may include task-based satisfaction measures, post-session satisfaction measures and other industry-standard or literature-recognized satisfaction measures (e.g., the Single Ease-of-use Question, System Usability Scale, Software Usability Measurement Inventory, etc.).
  1. The tester verifies all major test findings and the identified area(s) of improvements are reported.
  2. The tester verifies how effectiveness and efficiency were evaluated (NISTIR 7742 3.9 “Usability Metrics”).
  3. The tester verifies test results provided an analysis of the use, tested performance and error rates in order to identify risk prone errors -- with a potential likelihood of occurrence and adverse consequences (NISTIR 7742. results).
  4. The tester verifies the following NISTIR 7742 measures of effectiveness, efficiency, and satisfaction were collected for each participant:
    • Number of tasks successfully completed within the allotted time without assistance;
    • Time to complete the tasks;
    • Number and types of errors;
    • Path deviations;
    • Participant’s verbalizations; and
    • Participant’s satisfaction ratings of the system.

System Under Test Test Lab Verification
  1. The heath IT developer supplies the test scenarios used for the summative usability testing conducted on each of the safety-enhanced design criteria (specified in section (g)(3)(i)) submitted for testing. The test scenarios used in the summative testing should reflect prioritized use cases based upon a risk analysis.
  1. The tester verifies the existence of the test scenarios used for the summative usability testing, containing at a minimum, test scenarios to cover all of the safety-enhanced design criteria and associated capabilities (specified in (g)(3)(i)) submitted for testing/certification.
  2. The tester shall verify the name and version of the product are the final version (release) of the product for which certification is being sought.

Updated on 06-04-2024
Regulation Text
Regulation Text

§ 170.315 (g)(3) Safety-enhanced design—

  1. User-centered design processes must be applied to each capability technology includes that is specified in the following certification criteria: paragraphs (a)(1) through (5), (9) until the criterion's expiration date, and (14), (b)(2), (3) and (11) of this section.
  2. Number of test participants. A minimum of 10 test participants must be used for the testing of each capability identified in paragraph (g)(3)(i) of this section.
  3. One of the following must be submitted on the user-centered design processed used:
    1. Name, description and citation (URL and/or publication citation) for an industry or federal government standard.
    2. Name the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing user-centered design standards was impractical.
  4. The following information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied:
    1. Name and product version; date and location of the test; test environment; description of the intended users; and total number of participants;
    2. Description of participants, including: sex; age; education; occupation/role; professional experience; computer experience; and product experience;
    3. Description of the user tasks that were tested and association of each task to corresponding certification criteria;
    4. The specific metrics captured during the testing of each user task performed in (g)(3)(iv)(C) of this section, which must include: task success (%); task failures (%); task standard deviations (%); task performance time; and user satisfaction rating (based on a scale with 1 as very difficult and 5 as very easy) or an alternative acceptable user satisfaction measure;
    5. Test results for each task using the metrics identified above in paragraph (g)(3)(iv)(D) of this section; and
    6. Results and data analysis narrative, including: major test finding; effectiveness; efficiency; satisfaction; and areas for improvement.
  5. Submit test scenarios used in summative usability testing.
Standard(s) Referenced
Certification Dependencies

Design and Performance: This certification criterion was adopted at § 170.315(g)(3), and is required for all developers seeking certification to § 170.315(a)(1) through (5), a(9) until the criterion's expiration date, (a)(14), (b)(2), (b)(3) or (b)(11) of this section. 

Revision History
Version # Description of Change Version Date
1.0

Initial publication

03-11-2024
1.1

Removed resources listed under Standards Referenced as the information was already included the clarifications section regarding resources available for the UCD process.

05-22-2024
1.2

Expanded on clarifications under Paragraph (g)(3)(iv) regarding testing requirements for revised certification criterion(a) with added functionality.

06-04-2024

Certification Companion Guide: Safety-enhanced design

This Certification Companion Guide (CCG) is an informative document designed to assist with health IT product certification. The CCG is not a substitute for the requirements outlined in regulation and related ONC final rules. It extracts key portions of ONC final rules’ preambles and includes subsequent clarifying interpretations. To access the full context of regulatory intent please consult the Certification Regulations page for links to all ONC final rules or consult other regulatory references as noted. The CCG is for public use and should not be sold or redistributed.

The below table outlines whether this criterion has additional Maintenance of Certification dependencies, update requirements and/or eligibility for standards updates via SVAP. Review the Certification Dependencies and Required Update Deadline drop-downs above if this table indicates “yes” for any field.

 

Certification Requirements
Technical Explanations and Clarifications

Clarifications:

  • The application of user-centered design (UCD) during development and summative testing is limited to only those nine certification criteria specified in this certification criterion and only for which certification is sought, namely [80 FR 62670]:
    • § 170.315 (a)(1) Computerized provider order entry (CPOE) – medications
    • § 170.315 (a)(2) Computerized provider order entry (CPOE) – laboratory
    • § 170.315 (a)(3) Computerized provider order entry (CPOE) – diagnostic imaging
    • § 170.315 (a)(4) Drug-drug, drug-allergy interaction checks for CPOE 
    • § 170.315 (a)(5) Demographics
    • § 170.315 (a)(9) Clinical decision support (expires on January 1, 2025)
    • § 170.315 (a)(14) Implantable device list
    • § 170.315 (b)(2) Clinical information reconciliation and incorporation
    • § 170.315 (b)(3) Electronic prescribing
    • § 170.315 (b)(11) Decision support interventions
  • As a “revised” certification criterion, the safety-enhanced design (SED) certification criterion is not “gap certification eligible.” [80 FR 62609-62610, 62670] Thus, despite the fact that some of the functionality-based certification criteria referenced by the SED criterion are “gap certification eligible” for their functionality, all of the certification criteria referenced by this SED criterion (as applicable to certification scope sought) must have UCD processes applied and new summative usability test results as the basis for certification.
  • To demonstrate compliance with this certification criterion, UCD process(es) must have been applied to each capability of technology that is associated with the certification criteria named in this certification criterion. [77 FR 54188]
  • If technology is presented for certification and includes capabilities to which this certification criterion would apply, but for which certification is not sought, then those other capabilities for which certification is not sought would not have to have had UCD process(es) applied because they would be beyond the scope of certification. [77 FR 54188]
  • ONC-Authorized Certification Bodies (ONC-ACBs) should be notified when changes to user-interface aspects occur. ONC-ACBs are required to obtain a record of all updates to Certified Health IT Modules affecting the capabilities in certification criteria to which this “safety-enhanced design” criterion applies on a calendar quarterly basis. [80 FR 62727]
  • The documentation required by this “safety-enhanced design” criterion will become a component of the publicly available testing results on which a certification is based. [77 FR 54187]
  • ONC does not expect health IT developers to include trade secrets or proprietary information in these reports. [77 FR 54188]
  • The ISO definition of usability is “[t]he extent to which a product can be used by specified users to achieve specified goals with effectiveness, efficiency, and satisfaction in a specified context of use." [see also 77 FR 54186]
  • Health IT developers who have already followed UCD in previous development efforts for the certification criteria identified in the SED criterion would be performing a retrospective analysis for the purposes of certification. [see also 77 FR 54188] ONC notes that the discussion of retrospective analysis provided in the 2014 Edition Final Rule was in the context of health IT being certified for the first time to the new “SED” certification criterion. As an illustration of retrospective analysis for certification to the “SED” certification criterion, if a health IT developer had followed/applied a UCD process for any or all of the certification criteria referenced by the SED certification criterion, a developer would be permitted to cite that previously applied UCD process.
  • The Certification Program has modified § 170.315(g)(3) to reference the new certification criterion in § 170.315(b)(11). Certified Health IT Developers must assess user-facing functionality gaps between the requirements of § 170.315(a)(9) and § 170.315(b)(11) and, as necessary update, their safety-enhanced design (SED) testing. This means that functionality new to the (b)(11) DSI criterion, such as the functionality to modify source attributes and source attribute information at § 170.315(b)(11)(v)(B) and the functionality to
    enable users to provide feedback to evidence-based DSIs at § 170.315(b)(11)(ii)(C), would likely require user-centered design processes applied during development of those functionalities and included as part of summative testing.

Technical outcome – Developer must have applied UCD process(es) for any of the certification criteria specified in this criterion and for which certification is sought.

Clarifications:

  • The reported UCD process(es) must have been applied during the design and development of the capabilities/associated criteria.
  • If UCD had not been previously applied to capabilities associated with the certification criteria, the technology would ultimately need to have such UCD process(es) applied before it would be able to be certified. [77 FR 54187]
  • Examples of resources that technology developers may choose to review in order to select a UCD:
    • ISO 9241-11;
    • ISO 13407;
    • ISO 16982;
    • ISO/IEC 62366;
    • ISO 9241-210;
    • NISTIR 7741;
    • NISTIR 7742;
    • NISTIR 7804; and
    • NISTIR 7865 (“A Human Factors Guide to Enhance EHR Usability of Critical User Interactions when Supporting Pediatric Patient Care”).
  • Any UCD process selected by a health IT developer is appropriate, and it need not be listed in the examples we provided in order to be acceptable. [77 FR 54188]

Technical outcome – At a minimum, ten participants must have been included in the summative usability testing required for each required capability for which certification is sought.

Clarifications:

  • Although only 10 participants are required, health IT developers are strongly encouraged to exceed the mandatory minimum in an effort to identify and resolve more problems. [see also 80 FR 62671]
  • The cohort of users who are selected as participants will vary with the product and its intended users and should not be limited to clinicians but instead consist of test participants with the occupation and experience that aligns with the capability under testing. [see also 80 FR 62670 and 80 FR 62671]
  • ONC recommends that health IT developers follow NISTIR 7804 for human factors validation testing of the final product to be certified. [see also 80 FR 62671]
  • Recommended resources to consider:
    • NISTIR 7804; and
    • NISTIR 7741.

Technical outcome – The developer must submit the name, description and citation (URL and/or publication citation) for the industry or federal government standard used in UCD for the development of each required/applicable capability presented for certification.

Clarifications:

  • Examples of resources that developers may choose to review in order to select a UCD process include, but are not limited to:
    • ISO 9241-11;
    • ISO 13407;
    • ISO 16982;
    • ISO/IEC 62366;
    • ISO 9241-210;
    • NISTIR 7741;
    • NISTIR 7742;
    • NISTIR 7804; and
    • NISTIR 7865 (“A Human Factors Guide to Enhance EHR Usability of Critical User Interactions when Supporting Pediatric Patient Care”).

Technical outcome – If a non-standard UCD process was used in development (i.e., § 170.315(g)(3)(iii)(A) was NOT met), the health IT developer must report the process(es), provide an outline of the process(es), a short description of the process(es), and an explanation of the reason(s) why use of any of the existing UCD standards was impractical for each required/applicable capability presented for certification.

Clarifications:

  • No additional clarifications.

Technical outcome – The information specified in (g)(3)(iv)(A)-(F) must be submitted for each capability to which UCD processes were applied in product development and when summative usability testing was conducted.

Clarifications:

  • All of the data elements and sections specified must to be completed, including ‘‘major findings’’ and ‘‘areas for improvement.’’ [see also 80 FR 62670]
  • Health IT developers can perform many iterations of the usability testing, but the submission that is ultimately provided for summative usability testing and certification must be an expression of a final iteration. [see also 80 FR 62671]
  • Only lab based summative testing is necessary to be performed in order to demonstrate compliance with this certification criterion. Nothing precludes field testing and formative testing from also being performed and we encourage technology developers to do so. [see also 77 FR 54189]
  • Information must be submitted for each and every one of the criteria specified in the “SED” criterion to become part of the test results publicly available on the Certified Health IT Product List. [see also 80 FR 62725]
  • To demonstrate compliance with this certification criterion this information would need to be available to an ONC–ACB for review, but the form and format for how the data would be presented for testing would not necessarily need to be NISTIR 7742 template. This documentation would become a component of the publicly available testing results on which a certification is based. [see also 77 FR 54187]
  • For the requirement in (g)(3)(iv)(D), it is permissible to submit an alternative acceptable user satisfaction measure to meet the requirements of this criterion. As such, a health IT developer could meet the proposed NISTIR 7742 based approach for user satisfaction or provide documentation of an alternative acceptable user satisfaction measure. [see also 80 FR 62671]
  • It is important to note a specific distinction between paragraphs (g)(3)(iii) and (g)(3)(iv). Both paragraphs are complementary, but require different information and have different applicable timing considerations.
    • Paragraph (g)(3)(iii) requires the disclosure of the UCD processes that were applied and followed in the development of the applicable capabilities named in the “SED” certification criterion and for which certification is sought.
      • If a health IT developer had previously developed a capability following a certain UCD process, we clarified that the health IT developer would not need to pick a new UCD standard as a result of our rulemaking and use it to reengineer the capability following/reapplying that UCD standard. In that respect, the timing for this paragraph (g)(3)(iii) allows for retrospective attribution. [see also 77 FR 54188]
      • ONC notes that in the 2014 Edition final rule preamble we incorrectly referenced NISTIR 7742 on page 54188. NISTIR 7742 does not require the reporting of the UCD standard/process used. Rather, in the preamble of the 2014 Edition final rule we stated that requirement. [see also 77 FR 54188]
    • Paragraph (g)(3)(iv) requires that information/sections from NISTIR 7742 must be submitted for each capability to which user-centered design processes were applied.
      • In so doing, a health IT developer cannot use summative usability test results to meet this paragraph if such results pertain to certification criteria that have been revised with additional functionality and for which certification is sought.
      • In other words, if a health IT developer had followed a UCD process in 2010 to design its CPOE functionality, performed and provided summative usability test results in 2013 for the purposes of 2014 Edition certification, the health IT developer must produce a “fresh” set of summative usability test results for testing and certification. See also the clarification note about gap certification eligibility in the general clarification section above.
    • Once certified to the SED certification criterion, a developer can request inherited certified status for a newer version of its certified product consistent with 45 CFR 170.550(k). In so doing, the Certified Health IT Developer may attribute (“carry forward”) the summative usability test results created for the previously certified product version so long as the scope of referenced SED remains the same. In instances where an additional SED referenced certification criterion/criteria is/are added the summative usability test results applicable to those addition in scope would be needed.

Technical outcome – The test scenarios participants used during the summative usability testing must be submitted as part of the test results report.

Clarifications:

  • In accordance with NISTIR 7804 Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records (EUP) (page 8), we recommended that the test scenarios be based upon an analysis of critical use risks for patient safety, which can be mitigated or eliminated by improvements to the user interface design. [see also 80 FR 62670]
  • NIST recently developed an additional recommended resource for test scenarios: NISTIR 7804-1: Technical Evaluation, Testing, and Validation of the Usability of Electronic Health Records: Empirically Based Use Cases for Validating Safety-Enhanced Usability and Guidelines for Standardization.