Bulk Export AWS

Bulk Export can be used to export all or part of the FYI document store, including key metadata, to a designated Amazon Web Services (AWS) S3 directory. You do not need an AWS account to perform a Bulk Export, connection is made by entering the email address used to log into FYI, and a .pem file generated as part of the export process.

Using Bulk Export, you can export documents held in a specific Cabinet or to export documents for a specific Partner or Client. The data is held in a private S3 directory so you can connect and download files at any time.

Bulk Export could be used in the event of a practice split, or to export the documents for a specific client. 

Exporting all your documents would be used in the rare occasion that your practice chooses to leave FYI and it allows you to export documents in a well-structured fashion to then adopt a new system.

There is also a function to export to your practice OneDrive to the folder FYI - Export. Refer to Bulk Export OneDrive.

Important notes:

• Bulk Export is only available on demand. To run a Bulk Export, you need to contact the FYI Support Team to enable this function.
• Bulk Exports are only performed overnight between 6 pm - 6 am (depending on your location). Any processes that are still running at 6 am will be completed; all other requests will be delayed until 6 pm.

Index .csv File and Additional Metadata

As part of each Bulk Export, a report.csv file is automatically created with a summary of the files that were exported. The report.csv file is added to the root folder of the export e.g. /export-data/report.csv.

This includes the following columns:

Document Name
File Path
Modified FileName (the filename as exported)
Document Type
Category
Reference Number
Created On
Created By
Last Modified On
Last Modified By
Client Name
Client Code
Export Code
FYI Client ID
Cabinet
Year category

Note: 

• If you have included any additional columns in the export using the Configuration - Structure List, these columns are also included in the report.csv index file of the export (refer below to Step 1 - Set up the Configuration Structure for the Exported files).
• All categories will be included in the report.csv index file.
• If re-importing files to a new FYI, ensure that all report.csv files are included in the folder location 'export-data' for the Migrate App. This will allow this metadata to be applied to the imported documents.

If there is any additional metadata that you want to hold as a record as part of the export (such as Partner, Manager, Tags, Keywords) you can separately export a list from an FYI documents list as a record.
When you run an export, you can checkmark Include FYI Reference so that all the documents being exported have the FYI Reference Number appended to the end of the document name. By exporting a documents list in FYI with the Reference Number as the leading column, and including the columns for any additional metadata, you want to keep as a record, the FYI Reference Number can be used to cross-reference to the exported documents list (refer to Exporting Lists). Note that you can export up to 10,000 rows so you may need to export more than one list.

Notes on Bulk Export

• The Export is only available to FYI Admins.
• You can only have one instance of the Export running at one time.
• The Export list is finalised when a user clicks 'Create' in the Export wizard (end of Step 3 - Run the Export). Any changes to documents such as creating a new document or deleting a document, or changes to Client Lists, will not be included in the Export.
• If the category contains a backslash (\) the export converts this to an underscore (_). For example, if the category name is "GST\BAS" this is converted to "GST_BAS".
• We recommend planning your export to avoid coinciding with our weekly software release typically performed on Thursday evenings. A software release restarts the export process and can extend the time for the process to finish.

Please be aware of the following items included and excluded from the Bulk Export:

Security

Public and private keys are generated for you by FYI. Access to the exported files is secured by the key pair.

• The public key is held for you by FYI.
• The private key is linked to a username, which is your email address.
  The private key is generated as part of the Export as a .pem file. It is downloaded directly as part of running the Export in FYI.

Note: You must keep the username and the private key secure at all times as this provides access to your documents.

The Export security is controlled by an SFTP client (SSH File Transfer Protocol/Secure File Transfer Protocol). We recommend using Cyberduck for this. Another SFTP client can be used if you are familiar with SFTP clients or have one installed already. If you need more information on SFTP clients, refer to a reference source such as this Wikipedia article SSH File Transfer Protocol.

Steps when Running the Export

To run a Bulk Export, make sure you have contacted the FYI Support Team to enable this function. To run the export you will need to do the following:

• Step 1 - Set up the Configuration Structure for the Exported files
• Step 2 - As relevant, move Documents to be exported to a specific Cabinet, and/or check the user assigned as the Partner for the Documents
  
• Step 3 - Run the Export
• Step 4 - Download and Install the SFTP Client (Cyberduck)
• Step 5 - Run the SFTP Client (Cyberduck)
• Step 6 - Extract the Files from Cyberduck

Step 1 - Set up the Configuration Structure for the Exported files

The first step is to configure the structure of the folders that will be used for the exported files.

1. Log in to FYI as an FYI Admin.
   
   
2. From FYI, click Settings in the top right-hand corner.
   
   
3. Select Practice settings from the drop-down, then select Documents.
   
   
4. Go to the Export tab.
   This displays a log of the import and export functions that have been run or that are currently in progress.
   
   
5. Click the Configure button in the top right-hand corner.
   
   The Import/Export setup displays.
   
   
6. Go to the Export tab.
   
   

The Structure List determines the hierarchy that will be created by the Export. This determines the top-level folder and any sub-folders for the Export.

You can remove any metadata groupings from the Structure List and add any to it from the Available list. You can also change the sequence of the metadata in the Structure List.

1. Click to select what you want to move.
   
   
2. Drag and drop it either to the Structure List to include it, or drag it to the Available List to remove it from the Structure List. 
   
   
3. Drag it up or down in the Structure List to change the sequence.
   
   
4. Click Save.

You will also be prompted to check the Structure List and can make any changes before the export is run. Refer below to Step 3 - Running the Export.

Note: If documents have already been exported and you need to change the structure, this will not affect the folders in the previous export. Every export is saved to a separate folder and uses the structure as set when the export is run. The folder name of each export is set as the date and time when the export request was started.

   Return to Steps when Running the Export

Step 2 - As relevant, move Documents to be exported to a specific Cabinet, and/or check the user assigned as the Partner for the Documents

If you want to export documents for a specific Partner or Client, you have several options.

You can export documents for a specific Partner. This exports documents for clients where the selected user is held as the Partner for the Client. Include Partner as a column in the Clients list and in the Documents list to check the documents that will be exported if you are going to export using the selection for Partner.

You can also select a specific cabinet or all cabinets. If you are not exporting all documents, or not exporting for a specific Partner, ensure that the relevant documents are in a specific cabinet. You can move any documents using Bulk Update (refer to Bulk Updating of Filing Details).

To export documents for specific clients it's easier to run a full export, and then use your SFTP Client (for example, Cyberduck) to select the particular clients you require. Hold the Ctrl button to select multiple clients; refer to the instructions below on using an SFTP client.

   Return to Steps when Running the Export

Step 3 - Run the Export

When running the Export the first time, you download the private key, the .pem file.

1. 1. From FYI, click Settings in the top right-hand corner.
   2. Select Practice settings from the drop-down, then select Documents.
   3. Go to the Export tab.
   4. Click Run.
      The Export displays.
   5. Ensure the Medium is set to "AWS Transfer"
      
   6. As relevant, select the Partner, Client, Cabinet, or a combination of these.
      
      Selecting a Partner exports documents for clients where the selected user is held as the Partner for the Client.
      
      Select a Client from the drop-down to export all documents for that Client only.
      
      You can also select the Cabinet that holds the documents you want to export. To export all Cabinets leave this as "No option selected". This can be used instead of or in combination with Partner or Client.
      
      
   7. You can run the Export as Incremental so that it exports only documents since the export was last run, not the whole document system.
      
      
   8. When Include FYI Reference is check-marked, all the documents that are exported have the FYI Reference Number appended to the end of the document name. Refer above to Index .csv File and Additional Metadata about the index .csv file that is automatically created for each export. This includes the FYI Reference Number.
      
      If you need to keep a record of any additional metadata you want to keep as a record you can include the FYI Reference Number in the exported documents to cross-reference to the information in any additional exported documents list.
      
      
   9. The Include document threads option is ticked by default, to ensure that all documents in the thread will be included in the export. If you are importing this export into another FYI platform, these will be filed as separate documents. Untick the option if all documents in the thread are not required.

       

   10. Click Generate.
       This generates a .pem file with private keys.
       
   11. Click the link Your .pem file is ready. Click here to download it.
       
       
   12. Save the downloaded .pem file somewhere safe.
       
       
   13. Checkmark "I have downloaded the .pem file" to confirm you have done this.
       
       
   14. Click Next.
       
       The Export (step 2 / 2) displays. This shows the structure that has been set up. Check this is correct for the export you want to run.
       
       You can make any changes to apply to the export as required (refer above to Step 1 - Set up the Configuration Structure for the Exported files).
       
       
   15. Click Create to start the Export. 
       
       The Import/Export tab re-displays and shows the Export request. Any changes to Document or Client Lists after this point will not be included in the Export. 
       
       Note: The Cancel button displays which can be used to cancel the Export at this point. Once the Export request is picked up by FYI it will run until finished. When the Export is in progress, a grey progress bar displays below the Export request on the Import/Export tab
       
       Reminder: Bulk Exports are only performed overnight between 6 pm - 6 am (depending on your location). Any processes that are still running at 6 am will be completed; all other requests will be delayed until 6 pm.
       
       
   16. The export will show a green tick once it has been completed. 
       

       The export must be finished before moving onto Step 4 - Download and Install the SFTP Client (Cyberduck).

For any long filenames, the filename of an exported file will be truncated to 100 characters.

   Return to Steps when Running the Export

Step 4 - Download and Install the SFTP Client (Cyberduck)

Downloading and installing the SFTP Client is a one-off process. We recommend using Cyberduck and the examples below are shown using this as the SFTP to access the exported documents.

Click the link below to download and install Cyberduck.
https://cyberduck.io/download/

   Return to Steps when Running the ExportStep 5 - Run the SFTP Client (Cyberduck)

When you start the Cyberduck application, the following screen displays.

Navigate to Edit in the top ribbon and select Preferences.

1. Select Transfers and choose the 'Checksum' tab. Tick 'Verify checksum' in the Downloads section.
2. Select Connection and tick 'Enable debug log' in the Log section.

The Cyberduck set up is now complete and you are ready to start the connection process.

1. Click Open Connection.
2. Choose “SFTP (SSH File Transfer Protocol“ from the drop-down.
   
   
   
3. Enter the Server as per your region below:
   
   Australia and New Zealand:
   s-87445f28b4fb451ea.server.transfer.ap-southeast-2.amazonaws.com
   
   United Kingdom:
   s-411b4a77b93640ae8.server.transfer.eu-west-2.amazonaws.com
   
   Port - Do not change the Port number.
   
   
4. Enter the remaining details as follows:
   
   Username - Enter the full email that you use to log in to FYI. An email alias cannot be used to log into Cyberduck. This can be checked in the User Profile drawer. Refer to Managing Users.
   Password - Leave the Password field blank.
   SSH Private Key - Click Choose and select the .pem file provided to you by FYI. This file holds the private key.
   No changes are required under 'More Options'.
   
   

5. Click Connect.
   
   

6. If an Unknown Fingerprint pop-displays, click Allow.
   
   
   
   

7. When you log into the SFTP, you see a folder for the export with the folder name set as the date and time when the export request was started. Below this are the folders for the export structure that you specified.

   Return to Steps when Running the Export

Step 5 - Extract the Files from Cyberduck

The next step is to extract the files from Cyberduck.

The top-level folder will be displayed in the main viewing screen of Cyberduck. You can extract files from the top-level folder, or choose a sub-folder within it to display the folder you want to extract from.

Note: The Index .csv file is added to the root directory of the export, in the folder 'export-data'.

If you want to calculate the size of the export in AWS before extracting:

1. In Cyberduck, right-click over the folder you want to extract from.
2. Click Info.
   The following menu displays.
   
3. Click Calculate and let it run through all files to calculate the size of the export.
   This may take some time, depending on how many files are hosted.

To extract the files:

It is recommended to download the files to your local hard drive. From there, you can then save the files to your OneDrive or copy them to a portable USB or hard drive as required.
We do not recommend choosing OneDrive as the download location, the download may be slow and it may impact FYI syncing.

1. Right-click over the folder you want to extract from and select Download To.
   Tip: To select multiple folders hold the Ctrl button and click on each of the required folders
2. In the Browse For Folder, select the folder you want to save the files to.
3. Click OK and the files download to that location.

This leaves the data in Cyberduck after extraction, so they can be extracted again if required.

   Return to Steps when Running the Export

Running the Export Again

For any subsequent runs of the Export, you do not need to generate a private key again. The previously generated private key gives you access to the newly exported documents.

If you need to change the Export structure, this can be done from the Export Configuration (refer above to Step 1 - Set up the Configuration Structure for the Exported files) or can be changed when running the Export.

If a new export is started, the files will be saved to a new folder with the folder name of the date and time when the export request was started. If you re-export the same documents this will not overwrite any previous exports.

1. On the Export screen, make your selection for Medium.
2. If relevant, make a selection for Cabinet.
3. Click Next.

The Export structure is displayed, which can be changed if needed and you continue as above.

Note: If the keys need to be regenerated, users will need to run the export again starting at Step 1.

Accessing the STFP as a Different User

Only one user is authorised to access the SFTP at a time.

The .pem file is linked to the username which is the email of the person who originally raised the request.

If another user needs to access the SFTP temporarily, that person would need the .pem file and would need to use the original requestor's email as the username. They do not need the original requestor's password.

If another user needs permanent access, they can regenerate the private key as above with their username. After regenerating, the new user will have access to previous exports as well as any new exports. The original user will not have access to previous or new exports.

Re-Importing Exported Documents to FYI

If documents have been exported using Bulk Export AWS and need to be re-imported to FYI, they can be imported with the FYI Migrate app by selecting the Windows Explorer option. Refer to Using the FYI Migrate App to Sync your Data and Document Migration for Windows Explorer, HowNow or HandiSoft.

Note: Ensure that all report.csv files are included in the folder location 'export-data' for Migrate App. This will allow the metadata to be applied to the imported documents.

Troubleshooting

If you experience any issues during the export process, refer to Issues when bulk exporting using Cyberduck.