Whisk together lemon juice, oil, 1 teaspoon salt, 1/4 teaspoon pepper, and sugar. Stir in shrimp and caper berries. Refrigerate, stirring occasionally, at least 1 hour.
Meanwhile, stir together mayonnaise and garlic. Refrigerate until ready to use. Use a mandolin or a sharp knive to cut celery lengthwise into thin planks. Serve shrimp with garlic mayonnaise, shaved celery, crostini, and whole caper berries.