Bulk Export AWS

Bulk Export can be used to export all or part of the FYI document store, including key metadata, to a designated Amazon Web Services (AWS) S3 directory. You do not need an AWS account to perform a Bulk Export, the connection is made by entering the email address used to log into FYI, and a .pem file is generated as part of the export process.

Using Bulk Export, you can export documents held in a specific Cabinet or for a specific Partner or Client. The data is held in a private S3 directory so you can connect and download files at any time.

Bulk Export could be used in the event of a practice split, or to export the documents for a specific client. 

Exporting all your documents would be used in the rare occasion that your practice chooses to leave FYI and it allows you to export documents in a well-structured fashion to then adopt a new system.

There is also a function to export up to 500 documents to your practice's OneDrive folder FYI - Export. Refer to Bulk Export OneDrive.

Important notes:

• Bulk Export is only available on demand. To run a Bulk Export, you need to contact the FYI Support Team to enable this function.
• Bulk Exports are only performed overnight between 6 pm - 6 am (depending on your location). Any processes that are still running at 6 am will be completed; all other requests will be delayed until 6 pm.

Index .csv File and Additional Metadata

As part of each Bulk Export, a report.csv file is automatically created with a summary of the files that were exported. The report.csv file is added to the root folder of the export e.g. /export-data/report.csv.

This includes the following columns:

Document Name
File Path
Modified FileName (the filename as exported)
Document Type
Category
Reference Number
Created On
Created By
Last Modified On
Last Modified By
Client Name
Client Code
Export Code
FYI Client ID
Cabinet
Year category
Filing Status
Keywords
To
From
CC
Attachment

Note: 

• If you have included any additional columns in the export using the Configuration - Structure List, these columns are also included in the report.csv index file of the export (refer below to Step 1 - Set up the Configuration Structure for the Exported files).
• All categories will be included in the report.csv index file.
• Filing Status includes Filed and Archived documents only. Draft and Deleted documents are not included in the export. Refer to Notes on Bulk Export further below in this article.
• If re-importing files to a new FYI account:
  • Ensure that all report.csv files are included in the folder location 'export-data' for the Migrate App. This will allow this metadata to be applied to the imported documents.
  • Where an email contains an Attachment, this column will display a Yes value. When performing the re-import a paperclip icon will be displayed against the email in the list view in FYI. Related attachments can be accessed by exporting the migrated email. Refer to Exporting Documents.

If there is any additional metadata that you want to hold as a record as part of the export (such as Partner, Manager, Tags) you can separately export a list from an FYI documents list as a record.
When you run an export, you can checkmark Include FYI Reference so that all the documents being exported have the FYI Reference Number appended to the end of the document name. By exporting a documents list in FYI with the Reference Number as the leading column, and including the columns for any additional metadata, you want to keep as a record, the FYI Reference Number can be used to cross-reference to the exported documents list (refer to Exporting Lists). Note that you can export up to 10,000 rows so you may need to export more than one list.

Notes on Bulk Export

• The Export is only available to FYI Admins.
• You can only have one instance of the Export running at one time.
• The Export list is finalised when a user clicks 'Create' in the Export wizard (end of Step 3 - Run the Export). Any changes to documents such as creating a new document or deleting a document, or changes to Client Lists, will not be included in the Export.
• If the category contains a backslash (\) the export converts this to an underscore (_). For example, if the category name is "GST\BAS" this is converted to "GST_BAS".
• We recommend planning your export to avoid coinciding with our weekly software release typically performed on Thursday evenings. A software release restarts the export process and can extend the time for the process to finish.

Please be aware of the following items included and excluded from the Bulk Export:

Security

Public and private keys are generated for you by FYI. Access to the exported files is secured by the key pair.

• The public key is held for you by FYI.
• The private key is linked to a username, which is your email address.
  The private key is generated as part of the Export as a .pem file. It is downloaded directly as part of running the Export in FYI.

Note: You must keep the username and the private key secure at all times as this provides access to your documents.

The Export security is controlled by an SFTP client (SSH File Transfer Protocol/Secure File Transfer Protocol). We recommend using Cyberduck for this. Another SFTP client can be used if you are familiar with SFTP clients or have one installed already. If you need more information on SFTP clients, refer to a reference source such as this Wikipedia article SSH File Transfer Protocol.

Running the Export

To run a Bulk Export, make sure you have contacted the FYI Support Team to enable this function. 

Step 1 - Set up the Configuration Structure for the Exported files

The first step is to configure the structure of the folders that will be used for the exported files.

1. Log in to FYI as an FYI Admin.
   
   
2. From FYI, click Settings in the top right-hand corner.
   
   
3. Select Practice settings from the drop-down, then select Documents.
   
   
4. Go to the Export tab.
   This displays a log of the import and export functions that have been run or that are currently in progress.
   
   
5. Click the Configure button in the top right-hand corner.
   
   The Import/Export setup displays.
   
   
6. Go to the Export tab.
   

The Structure List determines the hierarchy that will be created by the Export. This determines the top-level folder and any sub-folders for the Export.

You can remove any metadata groupings from the Structure List, and add them from the Available list. You can also change the sequence of the metadata in the Structure List.

1. Click to select what you want to move.
   
   
2. Drag and drop it either to the Structure List to include it, or drag it to the Available List to remove it from the Structure List. 
   
   
3. Drag it up or down in the Structure List to change the sequence.
   
   
4. Click Save.

You will also be prompted to check the Structure List and can make any changes before the export is run. Refer below to Step 3 - Running the Export.

Note: If documents have already been exported and you need to change the structure, this will not affect the folders in the previous export. Every export is saved to a separate folder and uses the structure as set when the export is run. The folder name of each export is set as the date and time when the export request was started.

Step 2 - Move Documents to a specific Cabinet or check the assigned Partner (optional)

If you want to export documents for a specific Partner or Client, you have several options.

You can export documents for a specific Partner. This exports documents for clients where the selected user is held as the Partner for the Client. Include Partner as a column in the Clients list and in the Documents list to check the documents that will be exported if you are going to export using the selection for Partner.

You can also select a specific cabinet or all cabinets. If you are not exporting all documents, or not exporting for a specific Partner, ensure that the relevant documents are in a specific cabinet. You can move any documents using Bulk Update (refer to Bulk Updating of Filing Details).

To export documents for specific clients it's easier to run a full export, and then use your SFTP Client (for example, Cyberduck) to select the particular clients you require. Hold the Ctrl button to select multiple clients; refer to the instructions below on using an SFTP client.

Step 3 - Run the Export

When running the Export the first time, you download the private key, the .pem file.

1. 1. From FYI, click Settings in the top right-hand corner.
      
      
   2. Select Practice settings from the drop-down, then select Documents.
      
      
   3. Go to the Export tab.
      
      
   4. Click Run. The Export displays.
      
      
   5. Ensure the Medium is set to "AWS Transfer"
      
   6. As relevant, select the Partner, Client, Cabinet, or a combination of these.
      
      Selecting a Partner exports documents for clients where the selected user is held as the Partner for the Client.
      
      Select a Client from the drop-down to export all documents for that Client only.
      
      You can also select the Cabinet that holds the documents you want to export. To export all Cabinets leave this as "No option selected". This can be used instead of or in combination with Partner or Client.
      
      
   7. If preferred, tick the Incremental option so that it exports only documents since the export was last run, not the whole document system.
      
      
   8. When Include FYI Reference is check-marked, all the documents that are exported have the FYI Reference Number appended to the end of the document name. Refer above to Index .csv File and Additional Metadata about the index .csv file that is automatically created for each export. This includes the FYI Reference Number.
      
      If you need to keep a record of any additional metadata you want to keep as a record you can include the FYI Reference Number in the exported documents to cross-reference to the information in any additional exported documents list.
      
      
   9. The Include document threads option is ticked by default, to ensure that all documents in the thread will be included in the export. If you are importing this export into another FYI platform, these will be filed as separate documents. Untick the option if all documents in the thread are not required.
      
      
   10. Click Generate (or Regenerate Keys if an export was completed previously).
       This generates a .pem file with private keys.
       
   11. Click the link Your .pem file is ready. Click here to download it.
       
       
   12. Save the downloaded .pem file somewhere safe.
       
       
   13. Checkmark "I have downloaded the .pem file" to confirm you have done this.
       
       
   14. Click Next.
       
       
   15. If filters have not been selected, a message will be displayed when clicking Next to explain it may take longer to calculate the estimated document totals on the Summary screen (step 3 of the export). Click Yes to continue, or No to return to the Export Settings to select from the Filters.
       
   16. The Folder Structure will be displayed. Check this is correct for the export you want to run.
       
       You can make any changes to apply to the export as required (refer above to Step 1 - Set up the Configuration Structure for the Exported files).
       
   17. Click Next.
       
       
   18. A Summary of the export will be displayed, including an Estimated Document Count (including archived documents).
       
       Note: Where practices have a large number of files to be exported, the Estimated Document Count may time out, but the user will be able to proceed with the export.
       
   19. Click Create to request the Export. 
       
       The Import/Export tab re-displays and shows the Export request. Any changes to Document or Client Lists after this point will not be included in the Export. 
       
       Note: The Cancel button displays which can be used to cancel the Export at this point. Once the Export request is picked up by FYI it will run until finished. When the Export is in progress, a grey progress bar displays below the Export request on the Import/Export tab
       
       Reminder: Bulk Exports are only performed overnight between 6 pm - 6 am (depending on your location). Any processes that are still running at 6 am will be completed; all other requests will be delayed until 6 pm.
       
       
   20. A Pending entry will be added to Practice Activity to show the Export was requested. Click on the entry to view the details of the export, including the user, and the export settings.
       
   21. When the Export is being processed, the Practice Activity entry will be updated to In Progress. 
       
       
   22. The export will show a green tick once it has been completed. Click the arrow on the right to expand the section and view the Export Settings and Result.
       Note: The export must be finished before moving on to the next step.
       

Note: For any long filenames, the filename of an exported file will be truncated to 100 characters.

Step 4 - Download and Install the SFTP Client (Cyberduck)

Downloading and installing the SFTP Client is a one-off process. We recommend using Cyberduck and the examples below are shown using this as the SFTP to access the exported documents.

Click the link below to download and install Cyberduck.
https://cyberduck.io/download/

Step 5 - Run the SFTP Client (Cyberduck)

When you start the Cyberduck application, the following screen displays.

Navigate to Edit in the top ribbon and select Preferences.

1. Select Transfers from the menu icons and choose the Checksum tab. Tick Verify checksum in the Downloads section.
   
   
2. Select Connection from the menu icons and tick Enable debug log in the Log section.
   
   
3. Close the Preferences window.

The Cyberduck setup is now complete and you are ready to start the connection process.

1. Click Open Connection.
   
   
2. Choose SFTP (SSH File Transfer Protocol from the drop-down.
   
3. Enter the Server as per your region below:
   
   Australia and New Zealand:
   s-87445f28b4fb451ea.server.transfer.ap-southeast-2.amazonaws.com
   
   United Kingdom:
   s-411b4a77b93640ae8.server.transfer.eu-west-2.amazonaws.com
   
   Port - Do not change the Port number.
   
   
4. Enter the remaining details as follows:
   
   Username - Enter the full email that you use to log in to FYI. An email alias cannot be used to log into Cyberduck. This can be checked in the User Profile drawer. Refer to Managing Users.
   
   Password - Leave the Password field blank.
   
   SSH Private Key - Click Choose and select the .pem file downloaded from FYI. This file holds the private key.
   
   No changes are required under 'More Options'.
   
   

5. Click Connect.
   
   

6. If an Unknown Fingerprint pop-displays, click Allow.
   

7. When you log into the SFTP, you see a folder for the export with the folder name set as the date and time when the export request was started. Below this are the folders for the export structure that you specified.

Step 6 - Extract the Files from Cyberduck

The next step is to extract the files from Cyberduck.

The top-level folder will be displayed in the main viewing screen of Cyberduck. You can extract files from the top-level folder, or choose a sub-folder within it to display the folder you want to extract from.

Note: The Index .csv file is added to the root directory of the export, in the folder 'export-data'.

If you want to calculate the size of the export in AWS before extracting:

1. In Cyberduck, right-click over the folder you want to extract from.
   
   
2. Click Info.
   
   The following menu displays.
   
3. Click Calculate and let it run through all files to calculate the size of the export.
   This may take some time, depending on how many files are hosted.

To extract the files:

It is recommended to download the files to your local hard drive. We do not recommend selecting OneDrive as the download location, as the download may be slow and it may impact the FYI synchronisation.

Once the files have been downloaded, you can then move them to OneDrive, or copy them to a portable USB or hard drive as required.

1. Right-click over the folder you want to extract from and select Download To.
   Tip: To select multiple folders hold the Ctrl button and click on each of the required folders.
   
   
2. In the Browse For Folder window, select the folder you want to save the files to.
   
   
3. Click OK. The files will be downloaded to that location.

After the documents have been downloaded, the data remains in Cyberduck so the files can be extracted again if required.

Document Export Error CSV

After running the export, the number of documents that were unable to be exported will be displayed in the Export Results within FYI. For example, Draft emails will not be able to be exported.

In addition, a new file, errors.csv, will be automatically created with the details of each file that was unable to be exported. The errors.csv file is added to the root folder of the export e.g. /export-data/errors.csv.

This includes the following columns:

• Reference Number
• Document ID
• Key

To locate the documents in FYI, enter the Reference number from the spreadsheet into the Document List search field. Refer to Searching for Documents.

Running the Export Again

For any subsequent runs of the Export, you do not need to generate a private key again. The previously generated private key gives you access to the newly exported documents.

If you need to change the Export structure, this can be done from the Export Configuration (refer above to Step 1 - Set up the Configuration Structure for the Exported files) or can be changed when running the Export.

If a new export is started, the files will be saved to a new folder with the folder name of the date and time when the export request was started. If you re-export the same documents this will not overwrite any previous exports.

1. On the Export screen, make your selection for Medium.
   
   
2. If relevant, make a selection for Cabinet.
   
   
3. Click Next.

The Export structure is displayed, which can be changed if needed and you continue as above.

Note: If the keys need to be regenerated, users will need to run the export again starting at Step 1.

Accessing the STFP as a Different User

Only one user is authorised to access the SFTP at a time.

The .pem file is linked to the username which is the email of the person who originally raised the request.

If another user needs to access the SFTP temporarily, that person would need the .pem file and would need to use the original requestor's email as the username. They do not need the original requestor's password.

If another user needs permanent access, they can regenerate the private key as above with their username. After regenerating, the new user will have access to previous exports as well as any new exports. The original user will not have access to previous or new exports.

Re-Importing Exported Documents to FYI

If documents have been exported using Bulk Export AWS and need to be re-imported to FYI, they can be imported with the FYI Migrate app by selecting the Windows Explorer option. Refer to Using the FYI Migrate App to Sync your Data and Document Migration for Windows Explorer, HowNow or HandiSoft.

Note: Ensure that all report.csv files are included in the folder location 'export-data' for Migrate App. This will allow the metadata to be applied to the imported documents.

Troubleshooting

If you experience any issues during the export process, refer to Issues when bulk exporting using Cyberduck.