Spoiler: The key to open the 'hole' for the figs is not in your house. That key isn't needed yet. Have you created the ladder and retrieved memo #3? You will find the key near there.
